All Programming Language Suck? Just Build Your Own! – Language-Oriented Programming with Racket
Sperber, Mike
All languages have warts, wats, defects, and things that are just plain bad taste. Unless, of course, it's your language. Usually, implementing and maintaining your own language is a lot of work, but the Racket programming system makes creating a language as easy as having breakfast (almost) and thus a routine activity. Pick and assemble what you like from other languages, sprinkle your own favorite features, and voila - there's your fun experiment, your medium of expression, your educational language, your DSL. Racket's secret is its flexible syntax (hailing from Lisp), and its world-class macro system which is both super-powerful and easy to learn and use. (Going far beyond classic Lisp.) What's great is that you don't have to do all the work: Languages are just libraries of macros in Racket, and they can seamlessly interoperate with the Racket base language and each other. Don't worry if you dislike parentheses: Racket has you covered there, too.
 Hi there! This is a talk about creating your own programming language, specifically a language that doesn't suck. And for many of us, the language that does not suck is the first language that we learned. And for me, this was the language that ran on this computer here, the Apple 2e, which was a computer that my parents gave to me in 1983. And I just took that picture today, it still works, and it runs a language called Applesoft Basic. You can still find documentation on Applesoft Basic to this day. So here's for example, an introductory manual. And if you look inside that manual, you can see what a basic program looks like. You can see it consists of a bunch of lines. Every line has a line number, and on every line, there is a command. So it's none of that methods and classes and functions, nonsense that we have today. It's just straight up do this, do that, jump to that line, and it was a lot of fun during the day. So we're going to use the Racket system to implement that. And Racket is great. It's an entire toolbox that you can use to create your own languages. It comes with many languages. And it's just a lot of fun. And if you want to play with the code for this talk yourself, I put everything on GitHub in a repository called MikeSperber slash RC3. And that also links to a download link for Racket and has all the code for today. So anyway, I have to warn you, this talk is pretty heavy on code. So that's pretty much the only thing that I'm going to be doing is I'm going to write code in front of your eyes. And if that's just too much for you today, I'll understand and I won't be mad if you go see another talk. So but if you're in for some really serious hacking, then this might just be the talk for you. So Racket, when it starts up, comes up with a blank screen, such as this one was quite blank. Each file in Racket starts with a hash lang line that tells Racket what language that file is written in. And the system actually comes with many, many languages, one of which also happens to be called Racket. Now, Racket's okay language, but it also has some annoyances. So we could print something out by calling the display function. And if we run that program, well, down here, you can see the output. That's fine. But sometimes we also want to display several things, for example, a little bit of explanatory text along with a computation. So this year, well, you can see that the default Racket language is a Lisp dialect, which always has parentheses around compound forms, and has an operator always in front. So you don't write, you know, one plus one, you write the sum of one and one. Now we really, well, we hopefully we know what that should print out. But if we run that, we get an error message. And that's because display only takes a single argument to print out. And the second argument is supposed to be a port, which tells display where the output goes, if it goes to a file or the repl or whatever. So really, if we want, you know, several pieces of output, such as this, then we need to put in several calls to display. So there you go. Well, that's super annoying, right? And even, you know, Apple soft basic at the thing where you could just say print, and you could just put in several things. So you could do this thing that we were trying to do with display in the first place, right? But of course, you know, run this print, it's just not built in in the shape that we would like. So we're just going to define it ourselves. And to do that, we're going to write a macro, which means that the this form with print at the front is going to get translated into code that racket already knows about. And so we want to translate this into these two calls to display up here, right? And well, one restriction that we need to remember is that, you know, a form always translates into a single form. So when there's two forms here, in order to turn these into a single form, we just put begin at the front and parentheses around. So this is the translation that we want. I'll comment that out just so that we have that at our disposal. And in order to write a compile time function or a macro, we say define syntax, we're going to define a new piece of syntax. And that piece of syntax always starts with open parent print. And the compile time function that implements the macro always gets called with an argument, which is exactly the piece of syntax that the compiler saw. And we're going to call that form. Now, in order to translate it into this begin form, we need to take it apart a little bit, identify the arguments, we need to parse it. And for parsing, we will use a racket library called syntax parse. And I'm going to import that as a library. So it's not a default feature in the core of racket language. And since we were going to use syntax parse inside a function that runs at compile time or at syntax expansion time, as we like to say, we're going to say up here for syntax, so we can use it inside a define syntax form. So we write, you know, syntax parse here, we want to take apart this form thing. And we will write a pattern that describes the shape of the form. So we'll start with a very simple version of that pattern. So it starts with an open parent print, then there's an argument and that argument has to be an expression, which is denoted by putting colon expression there. And now we want to translate that into a different piece of syntax. In order to create syntax, what we do is I put in hash back quote, we want to translate that into display arc for now. So we'll just do a single argument version for starters. So I'll put that here, I'll run the code. So there's nothing in the REPL now, but now I can say at least Hello World, very exciting, and that works. But of course, it will only work with a single argument. We want to use it with several arguments. In order to do that, we can just say, well, we don't just want one arc, we want several. And to do that, we put three dots here. Almost looks too simple to be true. So it just says, well, there can be as many arguments as we want. And now, as we said above, we need to translate into a begin form. So we want to say like this, right? And then we want to have as many calls to display as there are arcs. And we can also do that by putting, you know, three dots here. And Racket will automatically figure out that that means we should have one call to display for each argument. And make sure that all the parentheses are closed. And let's try that out. One plus one equals plus like this. And sure enough, that works. So now we've added a new syntactic form to the Racket language. Pretty simple as it goes. And we'll use that as a small building block for our basic machinery that we're going to implement next. So, well, I guess it's a good time to save our work. Just to make sure that we're not losing anything, I'm just going to call this file basic dot racket. And there we go. So, okay, well, we got that print thing, which is similar to the print in basic, which is nice. But I don't know if you've ever used basic, it looks a basic program really looks like this, right? It says, you know, print hello world. And then you can see that there's a line number in the beginning, and then you go something like, yeah, go to 10. So it's always the first program that just print something repeatedly. So, yeah, well, this is obviously not there yet, right print, whether there's no line number. There's also still the funny parenthesis and tax. So if you create when you create a new language in racket, the way to do that is, well, you start with a parenthesis syntax. And later on top of that, we will implement the actual line based basic syntax that we might remember from our Apple two days. So, to that end, we'll just start by saying not, you know, 10 print, what we'll say open parent 10 print, you know, hello, world. Eventually, we're also going to have to take care of the lowercase print here, and then say 20, you know, go to 10, something like this. So that's what we're going to start with. Again, we have to think about how we will translate this into code that racket recognizes. And, well, you know, this program, it doesn't do anything yet. It just stores, you know, the print command, for example, it just stores that into into storage of the of the Apple soft interpreters, and I can then start the program by saying, well, I want to start the program at line 10, or something like that. So it's not enough to just sort of translate this into a call to print, we will also have to deal with the line numbers somehow so that also go to 10, for example, makes sense. So in order to store a piece of code, we can just give it a name and racket, right? So the idea is that we will translate it like this, we will say, well, we'll call we'll just define a function, we're just going to call that line 10. And we will make that one call print. And we will then define a function called line 20. And well, so go to of course, in racket doesn't exist, but we can just go to another function by calling it, right? So I'm just going to call line 10 here. And well, you can already see that eventually this this is hardly going to work as line 10 will never proceed to line 20. So eventually, we will also have to put a call to line 20 here, so that the code moves in the right direction. But we'll start simple, we're just going to start with a single line, and then we don't need to call that next function. How does that work? Well, we again say define syntax, right? And in this case, we're just going to define a piece of syntax that we're just going to call basic. Again, it takes an entire form as its argument, and we'll start just with a single command. So we're going to call syntax powers here, pass it the form and say, well, basic. And the way basic commands work is there's a line number beginning, which is an integer. And then there is a command, which is in racket terms, it's just an expression. So, well, what do we need to do? Again, we need to translate it into another piece of syntax. So, well, we can see up there, right? We want to translate it into define. And then, well, there's an open parent, we're going to have to cook up that line 10 name. Also, we'll defer that problem for a little later. And we will just insert in here the command in the body of the function. So, okay, so that leaves the problem of computing the name line 10 from the line. So we want to compute that name by sticking that, you know, line minus in front. And to that end, well, I want to call a function called format it and format it. Well, so it takes something that I'll explain in just a moment. It takes a pattern, which is sort of like printf, except, well, the pattern language is slightly different. So the tilde a says we should insert something in here and we want to insert the line number. Now, I'm kind of tempted to put in line number here. Now, I need you to understand one thing is that, you know, as syntax, a piece of syntax that represents a number is not the same thing as the number itself. So, for example, if the line number is 10, you know, down here in the REPL, I can type 10, I get 10 as a result. But if I type this funny syntax thing, then it says, well, there's a piece of syntax and it contains that number 10 in there. If we want to stick it into the name, we really need the number 10 in order to extract the sort of the actual piece of data that is represented by the syntax, I can call a function called syntax E. So if I call syntax E of, you know, here, hashback quote 10, then it gives me that 10. So really, you know, that's what that line number is. So I need to call syntax E here. Moreover, you can see here that I left three dots here, format it also takes an argument that is called the syntactic context. Now, the macro system in Racket has great sophistication when it comes to identifiers. If you've seen macro systems in other languages, such as in C, they're very simple about names. And you can introduce name clashes, very easily and Racket avoids that. That's not going to be the main subject of this talk. So suffice it to say that as the context, we want to define the line numbers are at the same level as the basic form that appeared. So we'll just stick this here. So just a little piece of magic here that we're just going to have to accept for the moment. So well, this thing, we want to stick that in here, right in here where the three dots are. And to that end, I will just give it a name. I'll call that thing name. And I want to stick it in here. Now you have to understand a very subtle issue here, which is that, so for example, you know, when I have a pattern variable here that appears in the pattern, I can just stick that in the syntax. For example, here I can just put in command and Racket will figure out that I really meant this command here. Unfortunately, well, I mean, there are reasons, but this, that does not work for this, which is a regular binding. It's not a pattern binding. In order to tell Racket to, well, you know, stick the result of evaluating that piece of code here, I will need to put in hash comma name, which says, well, here's an expression, you know, stick the value of evaluating that expression in here. So that's, I think that's confusing at first, but you'll get used to it. So remember, when there's something in here in the pattern, you can just stick it in here. And if you have regular bindings, or if you have a piece of code that you want to stick in your syntax, you will need to prefix that with a hash comma, which is also called unquote. So let's see if that works. We'll run that. See here, I already ran into the first trap here that I just told you about. I called syntax E on the line number, but the same kind of kind of goes in reverse. So if I can't just use a pattern variable in regular code, instead of instead, what I need to do is I need to prefix that pattern variable with hash back quote. Okay. So and then it knows, okay, I need to look among the pattern variables for this. So if I run this, well, then it says format it, it doesn't know about that, and that's because format it in racket is in a separate library. So I'm going to need to jump upwards a little bit and say here that for syntax, I don't just want syntax powers. I also want a library called racket syntax, and that hopefully contains format it, go back to that code, press run, and now at least it's silent. And now, well, I could try doing that by saying basic, you know, 10, you know, print Hello world. And doesn't do anything, right, as it should, it's just to just stash that piece of code. And but if I now type this, it executes that first line of code. So hooray, we've got a very basic, basic version of basic here. That works. Okay, so now we have a single command and a basic program. But of course, we want to have several of them. And so up here, we already know how to do this, or at least how to tell syntax powers about this, we just put three dots here and says there can now be several lines in a, in a basic form. So, but of course, we need to translate that differently. Right now, our expansion just has a single definition for a single line. And now we need to expand that to do several definitions. So to that end, we will record as a functional programming language, we do a map. So we apply a function to all the line numbers and commands, and the function always returns a piece of syntax for a single definition. Let's try that. So I'm going to say lambda here. I'm going to put three dots here, because there's a subtly that I'll explain in just a moment. And now the question is, what do we map over? Well, we want to map over all, you know, all the line numbers and all the commands, of course. So if we have line number here, well, you know, it's not just one line number, right? It's several line numbers, one for each line in the program. So we always have to use it like this, right, that we always put the three dots behind it. Otherwise, racket will complain. And moreover, line number again is a pattern variable that comes from this line in the syntax parse form. So, you know, the pattern variables, they really only work in this hash bang, this hash back quote form. So that's what we need to write. And also we do the same thing with the commands. And we're still not done because, well, you remember, you know, hash back quote returns a syntax object. But what we really need is a list because we're feeding that into map. So racket has a handy function that does that called syntax to list. Also, here is syntax to list for the commands. And that allows us to map over, well, here, the line numbers and the commands. Now, you know, now here's the subtly. Suddenly, because we now have line number and command be regular lambda parameters, they are no longer pattern variables. Okay, they're just regular variables. And this, I think, really is the most subtle aspect of the, of the racket macro system is sort of, you understand this issue, right, the pattern variables come from up here. And anything, everything else, everything that is not here in a syntax parse form in the, in the pattern part is not a pattern variable. So in this case, these are now regular variables. And that means they should not appear in hash back quote anymore. In this part where the line number is, and also means that a racket is not just going to replace the pattern variable by the expansion down here with commands. So I've got to push hash comma here. So if you figure that out, if you figure out that distinction, I think you'll be fine. So let's try that out. I'll run the program. Thankfully, there are no error messages. Let's try again that basic program. Oh, it complains again, there's another trap. Well, not a trap. Really, it's something that we've seen before is map returns a list. And you would see here. So it says your quote open for N. That means it returned a list. And that means, well, map produced a list, but racket really expects a syntax object. You know, if you're coming from lists or from closure, you expect sort of lists to be usable as syntax objects, but racket makes a strict distinction here. And it uses that distinction to track source code locations to give you good error messages and also to track hygiene to keep track of lexical binding, which is not the main focus of the tutorial, but just so you know. So anyway, we need to definitely stick hash back quote here. And now we have several definitions. And because we have several definitions that we want to stick in one single form, we use begin just like we did with print before, just going to indent everything nicely. And of course, now we want to stick the result of evaluating this in here. So I've got to put hash comma here. Okay. And unfortunately, there's still another subtly. If you look at the expansion here, you can see that it would, well, hopefully you can see that it would expand into something like this, right? You have begin, you know, from here. And then you have the open paren, which is from the list produced by map. And then inside those open paren's are the definitions of the various lines, like, you know, line 10, line 20, and so on. And you can see that there's one pair of parentheses too many. So we really don't want the racket macro expander to stick that list in there. We want it to stick the elements of the list one by one inside the begin. And to do that, we need to put a magic character here, also sometimes pronounced unspicing. So and that does exactly what we want. So it's just like hash comma, but it expects a list and it will stick the elements of the list there effectively removing one pair of parentheses. So let's try that again. Oops. So here's that basic form. At least it doesn't error out anymore. Let's see if we can start it. And it says, Hello world. And we could also now extend it, right? And put another line here. So right now we only know print. So do that. And so we can start line 10. And you can see that it, well, line 10 runs, but really we expect the program to go on to line 20 after line 10. Line 20 is there, but we haven't yet linked the lines together. So that's going to be our next problem. So what we need to do here is, well, after the command, you know, associate with the line happens, we need to stick in a call to the next line, right? Now, how do we get that call, right? In order to produce the call for the next line, we need the line number of the next line. But we don't have it here. We just have the current line number. And to do that, we're just going to map over an additional list, which is going to be like the original list of line numbers, but we'll shift it by one. So along with the current line number, there's always the next line number. And to that end, I'm going to call the list function, CUDR, which just, you know, removes the first element from the list. And so that gives us that. And now, of course, map would complain because the lists do not no longer have the same length, right? This list is one shorter. And so we need to append an element to make up for the last element. And since there is no line after the last line, I'm just going to pass false, right? This is a racket syntax for false hash f. And we'll need to make sure that after the last line, we don't try to stick in a call. Okay. So here we are. We need to, of course, make sure that now that we're mapping over three lists and not just one, I need to add an additional parameter here, right? And now we could say call next line. And I could just, you know, stick in a call to format it like this, right? And then pass next line number. But that would be poor abstraction. And well, it's kind of copy paste code. So we could pull it out into its own function definition. That's a useful exercise. So I'm going to take that code, copy it one last time, stick it up here. Now there's two subtleties. Let me get rid of this obsolete comment here that we have to pay attention to. So I'm going to call that function, you know, make line name. And of course, we need to pass in the line number, this thing here. But we need to pass in one more thing because this basic here that you see here, remember that it referred to this basic down here, right? It used to be here. But that's no longer in scope because we're outside of its parentheses. So we also need to pass it in as an argument. And we'll call that, it's called the context, used by Rackett to make sure that the definitions appear in the right place and are referable by their name. Okay, so we have this great. And then we can take out those two calls here and call make line name instead. Remember that we now need to call this basic thing that was there before and line number. And the same thing over here. I'm sorry, it's called make line name, probably noticed. And we'll do the same thing here, call it make line name, remove all the crud here, let's duplicate it. And now we have a nice call. Of course, well, we'll get to the fact that it's not a call yet in a moment. But also, I want you to think about one more aspect of MaxRexpansion. This definition up here of make line name is just a regular procedure definition. But we want to call it from the macro expansion process. So essentially at compile time, we're a macro expansion time. But this is just a regular runtime function. It will not be available until runtime happens. And in order to tell Rackett, no, this is a function, please make it available to macro expansion. We need to replace the define by define for syntax. So define it. So it's available during syntax expansion. Okay, so one more thing. This is of course not a call yet. In order to have a call, we need to have parentheses to the parentheses around the name of the next line. So I'm going to put do this and put parentheses around it and then stick this in with hash comma, right. And that would be the call to the next line. Well, except it isn't quite because remember, next line number can be false, right. And then we would not have the then then we would not have a function to call there. So we really need to make a case distinction here. I'm going to say, well, if there is a next line number, then this is a great function to call. This is the function of the next line. And if there isn't, well, I'm just going to call void, which is a function that does absolutely nothing that's built into Rackett. Okay, let's try it out. Let's give it a whirl. Well, oh, it says next line number, reference to an unbound identifier. That's because I put a type over here. Let's try that again. Okay, now it at least goes through. Let's see, we had that program that had two lines in it, doesn't error out. That's great. Call line 10. And, ah, okay, now it runs both lines. That's wonderful. But we see that we made a small mistake in the implementation of print, at least in Apple South Basic, when you call print, it prints return a new line at the end. So why don't as a last act of this particular task, we go up here and stick up here, they called a new line. That should work. Try that again. And now we get the output as we've wanted it from the beginning. So now we can have a sort of a basic program that has several lines where the lines are run consecutively. That's great. So what's next? Well, here we can already see what might be next, which is the infamous go to command in basic, which just jumps to a to another place with no memory of where it came from, or what was going on beforehand. So if we look at, you know, our code here, we can see that go to is going to require special treatment. We can't just implement it the way that we implement it, that we implement print, because of course, when there's a go to, there should not be a call to the next line after the go to that would make no sense because go to just transfers control directly. Now, well, things are getting a little bit unwieldy at this point in this in this longest function here. So I will put the code that translates a single command into a separate function, and we'll see how that goes. So here, again, remember that we need to say define for syntax, so it's not a macro, it's a function that is called from a macro definition, going to call that function translate command. And well, so we pass in the, we pass in the actual command. So that's the, that's, if you will, the original source code from the basic program. And we need to translate that into a racket code. And well, and the idea is that we use translate command to produce the body of this function here. So, of course, that means we need to also pass in call next line. So let's see, let's pass that in. Great. And for now, we're just going to keep that very simple. We're just going to, going to keep the definition from up here. Now, again, remember that when we produce syntax, we can only produce a single form at a time. So there are two forms here, but we know we can always combine two forms into one by sticking a begin in front. So we'll just do the abstraction for now without adding additional functionality. So instead, we will say, okay, translate command, command, and call next line and omit this to a way with that, make sure that all the parentheses line up. Yeah, it's getting, getting to be a lot of parentheses and see if that works. So run that and, you know, program runs as before. That's great. So we want to give go to special treatment. And in order to do that, well, we need to inspect the command here and see, well, with, if it's a go to command, then we give it this treatment. In all other cases, we just translate to the code that we already have. So I call syntax powers, call pass the command. And now I put here a pattern where it says, well, if it's a go to command, looks like this, go to, there's a line number and that line number is an integer, then I'm going to do one thing. And in all other cases, I want to do something else. And you can see here that, you know, this is now a syntax parse that has several clauses, and it just tries them in order with the compiler. While it does syntax expansion, first tries the first one, then the second one, the first one that matches gets to be run. And this thing here is just a wildcard pattern that matches anything that you want. So that's the fall through clause at the end. Okay, so in this case, well, we have go to now, here's another subtly, in order, we really want the word go to to appear there. But remember that syntax parse does pattern matching. So whenever there's a name here, that name is just going to match whatever is there in the input. It's really a subtle issue. And we really want the word go to to appear here. And another subtle issue is that we want that to be consistent with the rest of the binding structure of the program. We'll ignore that here. But in order to do that, we need to put something like this, a magic clause here at the beginning that says, well, go to whenever it appears here, means that we really want the word go to to show up there. So for all of our specially treated commands, we need to put that in, we put them, you need to put the name of that command into the literals clause up here. Okay, so now, when there's a go to, we don't, you know, we just want to generate a call to the procedure that's associated with that line number. And we get the name of that procedure by saying make line name. Well, and remember, make line name now needs that context argument up here, and we don't have it here. So we need to add another parameter, call that context up here, we pass basic. And okay, got that here. And then we can call make line name with context and the line number. Okay, so got that here. And that's just the name of the procedure, we still need to make a piece of syntax with a call to it. So we put hash bang back quote, put a pair of parentheses. And in the middle, we need to put hash bang comma to actually stick that name there. Okay, well, let's see if it does anything. Oh, error message. Well, again, I fell into the same trap that I always fall into line number here is a pattern variable. So it really is only valid inside a hash bang back quote. Well, it is inside a hash bang back quote, but the hash bang comma kind of undoes that it's a, so we refer to it as a regular variable here, in order to refer to a pattern variable. Again, we need to put hash bang back quote here. So it must seem confusing at this point. But if you just do a bunch of stuff with macros, you will soon get the hang of that, I think, and then you'll recognize the error message right here, it says pattern variable cannot be used outside of a template. So line number is a pattern variable. Now we can see, oh, that nicely, that occurrence here of line number nicely refers to that one. Let's see if it runs now. Okay, see if we can do this. So, well, here's a rule that I forgot about, but fortunately, it's easy to remember because there's an error message says literal is unbound in phase zero, right? Phase zero has to do with the phase zero's runtime. Don't have to worry about that too much, but it just means that go to has no definition that it refers to. And we're not interested in a definition because we're translating calls to go to into something else anyway. So I'm just going to say put in a dummy definition here to false. And that just means if there's any reference to that go to, then, you know, syntax parse will recognize it. Let's push the wrong button there. Let me try again. And, well, at least it doesn't error out. Let's see if we can run the program. And that's pretty neat. And now, well, I guess it's time that I put an actual basic program here into the source code so I don't have to keep retyping it like this. You know, like this, for example, and how are we going to see that it works? I'm just going to put a line 20 there that says go to 40, make that line 30, and put line 40 which says, you know, the end, right, so that we know the program is done. Let's see if that works. We could call line 10. And indeed, it says how the world the end and it skips hello again. So go to works now. All right, what's next? Well, we can just keep adding commands, right? But of course, some cause more trouble than others. And one that causes a little bit of trouble is assignment, right? We can have variables in basic might look like, well, in basic it would look like this, right? A equals 42. And, you know, that would establish a variable called a, and we could then use it, right? We could say something like this. And I'm going to use a syntax slightly different. So of course, it has to be parenthesized, the operator has to be in front. So might look like this. And I'm going to use colon equals to avoid confusion with the with the equals sign without a colon, which is the equality operator. Okay. So we'll just use the same technique that we used before. So I'm going to just like go to right needs to be a literal, we want another case to match only when there's an actual colon equals sign there. Well, here's a variable. And that has to be an identifier. That's how we talk a tell syntax parts that needs to be an identifier. And then there's a right hand side, which can be any expression that we want. Okay. So got this here. And now we just need to generate a piece of code that works with it, right? And in, in racket, variable assignment happens with an operator that's called set bang. So the exclamation mark is pronounced bang, to denote that there's an evil side effect happening here. So, and we can just use these two pattern variables, we can just use the right hand side here. And, okay, well, remember that trick that we needed here with go to we need it with colon equals to so we need to provide a dummy definition. And now, well, here's a little problem, right? It says, a is an unbound identifier down here. And that's because, well, racket distinguishes between binding an identifier. So you're establishing that the identifier exists and then changing the value in the storage cell associated with that identifier. Those are two separate things, right? And so we could make that error message go away by saying, you know, define a to some dummy value. But then of course, that's no longer basic, right? It's, if we would need that additional definition in basic, we can just introduce an identifier by doing an assignment to it. So somehow we need to generate the set of variables that appear in assignments and make sure that we also generate definitions for those identifiers, dummy definitions at the beginning. So, well, we need to generate those definitions. And for generating those definitions, we need the set of variables that are in a program. We'll start by just collecting the variables that are in a single command. And to that end, I'm just going to define a new function here, call that collect variables. And I'm going to pass in the command. Okay. And well, not just the command, but we will want to collect a list or rather a set of those variables, because of course, a variable might occur in several assignments. So we will use something called an it set. And we'll define that in a minute. So a racket comes with a library that allows you to manage sets of these identifiers. And we will need to import that library up here. So call that syntax, that's called syntax it set. And that allows us to deal with those sets. Okay. And of course, we need to look inside those commands. So we're just going to use the same syntax part essentially the same syntax parts that we had up here, right? So we have various literals here with go to, of course, there can't be any variables introduced by a go to, we're mainly interested in the clause that has to do with assignments. And well, how does that work? So in that case, what we need to do is we need to say, free it set at and it set and add the variable here. And anything else, well, we're not going to do anything. I'm just going to call void the function, the built in function that doesn't do anything. Okay. Okay. And now we need to call that function collect variables for each command. And we need to do that up here, from the basic form, right, where we have all the commands at our disposal. So first of all, we need to create an empty it set that we can add to call it it set. And so we're going to call a function, it's got a cumbersome name mutable, it's set. Oh, and not just that mutable free it set, because again, racket makes subtle distinction between identifiers, depending on whether they're free or bound, don't worry about that. This, this is going to work. And then what we do is, well, we want to call collect variables on each command and passing in that it set. And but remember, we don't just have a single command, we have a whole list of them. And so we need to do this. Well, we take the command dot dot dot, and of course, need to wrap that hash back around it, convert that into a list, we've done that before. And then we can use a built in function called for each, which just calls this function on every argument on every element of that list, right? It's sort of like map, but that returns your list of the result, whereas for each just doesn't bother with the results and throws them away. So now we have a set of identifiers. We just need to generate definitions from it. And these definitions need to go here, right, before the actual code for the basic program. So again, we can use map, right, to map all over all the variables. We'll figure out in a moment what we're going to do with those. And well, we'd like to map over it set. Unfortunately, it set is not a list and map once a list. So we are going to use a function called free, it's set to list to make it a list, right? And so for each variable, we want to generate a dummy definition. Again, we need to generate a piece of syntax says define, we stick that variable in, and we'll just define it to false to make sure that things don't get mixed up with regular basic values. Okay, now remember, again, I mean, you can see here the other called a map, right, we want to make sure that this gets spliced into that form here. So again, I call that I use that same funny magic thing here, hash comma at, which splices all those definitions into that begin form. Okay, well, let's see if that works. And now it says, oh, that variable is already defined. That's a good sign, because you can't have several definitions for a single variable. So we put one in ourselves. I'm going to delete that. And okay. And now we can see that there's a variable here, right? It's bound to false as it should be. And let's try running the program. Well, not much happens. Why is that, right? Why does the program not produce anything? Well, we could look at a is now 42. Let's see, let's go up here. And we can see here that translation of colon equals and you see, you can see I forgot something. This just sets the variable, but it does not go on with the program as the rest does. We need to put a begin here and then put in the call to the next line. Let's see if that works. And now line five. And that actually prints out the value of that variable. So now we've got variables in our program. That's wonderful. Next up are conditionals in basic. So how does that work? Well, basic has a statement called the if statement might look like this, right? We compare a with 42. It's kind of silly. We know what the result is, but just to demonstrate the feature. And then depending on whether that's true or false, we could whatever said B to one or set B to two. And we'll just print that out along with the A here, just so we know what came out. All right, well, we need to translate the if command, of course. So we need to go up here into our translate command function. We need to add if here to the list of literals. So we can recognize ifs. And it looks like this. Well, we have some tests. That's an expression. We have a then branch. That's an expression. And we have an else branch. That's an expression. And well, we're just going to translate it into brackets if, but we need to make sure that the then branch and the else branch also get translated. So we use hashback quote. Again, you know, this is not the same if as here. This is now brackets if we're just going to leave test as it is. And then we call translate command. And remember, we need we want the result of translate command stuck in there. So we need to put hash comma in front, pass in some context. You know, again, you know, we need to pass pattern variable here. So we need to put hashback quote in front and we'll pass call next line as before. And do the same thing here, translate command context and call call next line. Okay, well, let's try running it. That doesn't work, right? We get an error message says B is unbound. Why is that? Well, we're generating definitions for all the variables that occur. But we but so far, we're only looking at assignments that occur at the top level, we're not looking inside an if so we need to make sure that we also extend collect variables in the same way that we extended translate command. So we also need to add that to the list of literals here and just put the same, we can just copy the pattern from up here. Copy that here. And well, we just need to call collect variables recursively on the two branches, right? So we call collect variables on then with the it set. And we call collect variables with the else on the it set. Well, let's see if it gets any better. Well, at least it doesn't air out. Let's show up our program again, we could call it starting at line number five. And that's not bad. So we now see the it says Princeton number one, which comes from this assignment here, right? And so B is assigned to one if A is 42, when A is indeed 42, we could also set it to 41. Try that again, line five. And we see that it now prints a two because the branch went the other way. I'll just change that back and save. Great. So we'll add one more feature. That's actually not just one command, but we'll need three commands. And that is basic sub routines. So what could that look like? For example, in line 1000, we could say we're just going to print the B variable. And then in line 1010, we just put a return statement. So Applesoft basic doesn't have functions in the way that we know them, or methods, it just has these sub routines. And so we could call, we could output B by just saying, you know, whatever line eight here, and we could say, go sub line 1000 like this. And the idea is then that it will go and print B, right? And return and go on after that with Hello World and whatnot. We could also call it several times, of course. Okay. And yeah, we'll just leave it like that, two commands. Okay, so go sub and return. What should we do? Well, we go up here. And obviously, we need to do something with translate command, we will just add go sub and return. And these are not built into rackets. So we need to have go sub false dummy definitions for them to do that. And so how does that work? Well, go sub is kind of like go to accept the program goes on, right? So while there's go sub line number, which is an integer. And of course, well, we need to generate a call, just as we do with go to, right? So going to put on just going to duplicate that from up here. And the only difference is after that call is done, right? When that finishes, then after that is done, we need to continue with the next line. Unfortunately, we know how to do that because we've got the call to the next line sitting up here. So I just stick call next line here. So that's the only difference between go to and go sub. And of course, we also need to implement return. And well, what happens with return? With return, nothing happens. Also return is similar to go to and goes up and that it doesn't go to the next line, it just returns. And when there's an active go to, it will just go back to the line after that, it will go to that call to call next line here. Let's see if that works. Okay, so at least we don't get an error message line five. And well, how do we interpret that output? I kind of forgot what the program was. Well, here it said goes up 1000, it printed B, and then it went on with the program as before. Arguably, the program should finish here after line 40 and before line 1000, but I'll leave that for you as an exercise to implement. So now we have enough basic there to at least give us a fair idea of how to do this whole thing and maybe how to add the rest. And I promise that we would also get actual basic syntax and we only have a few minutes left. So let's get to it. In order to do that, we need to make sure that the functionality that we've implemented in this file is available in other files. And so we need to add what's called a provide form. That's like an export or public annotation or something like that. And we just need to provide the basic form that we implemented, the print command that we implemented, go to, you know, all those things that we added to implement our basic syntax return. There we go and save that file and then we're done with that part. Now for the actual basic syntax, there's no way around writing a proper parser and we don't have nearly enough time to do that. It's not that hard, but it takes a little bit of time. So I've prepared the code to do that. It's just called a reader. So here's a file called basic reader dot racket. And if I run that, well, you can see here, for example, there's a function called parse line takes two arguments, one of them is called Zerts. That's just context information about what file this code resides in and so on. I'm just going to pass false here. Later that value will be provided by the racket system. And the other argument is an input port, which is just a reference to an open file or something like that. But we can also use a string by just saying open input string here. So if I say, you know, 10 print a plus one and call that function, you can see that it returns list structure that corresponds exactly to, you know, the syntax that we, the parenthesis syntax that we implemented earlier. We just need to hook this up to the rest of the racket system. And for that, we need to write a little boilerplate function going to call that basic read syntax also takes a Zerts and an in argument. And that function calls parse program, which does the same thing as parse line, but for an entire program do this. And we want to embed the result of that in something that racket will later understand. So do that. And we need to create a module declaration for a racket. I'm just going to call that module basic. My name doesn't particularly matter. It imports the racket language. And it also requires so it imports the code that we've just written, which is in the racket dot arc, KT file. And now here's the result of parsing the program, we still need to put a basic form around it and splice in the result of parsing the program. So that's that. Now you might have noticed that the back quote up here and the comma at here doesn't have a hash with it. It doesn't create syntax structure, it just creates an ordinary racket list structure. We still need to convert it to a syntax object. And for that, we use a function called datum to syntax and just pass false here for the context. That's enough. Make sure that all the parentheses line up and then we're good to go. Well, almost good to go. We still need to export this function and we need to have a different slightly different name for it. We're just going to call it read syntax on export. And that's a magic name that racket will later recognize. So I'll save that file and then we are good to go. So in order to try this out, I've prepared another little example file called basic demo syntax dot rkt. This is what it looks like. And you can see the hash line line up here declares that we want to use the reader that we just implemented. Basic reader dot rkt. And what follows is a very much Apple Soft looking program. And I can run that by just typing line 10, like we used to do that. And we can see that it pretty much works. So there you have it. I mean Apple Soft is just about as different from usual racket as it can be. And so if you can do that, then you can implement just about any language you like with the racket system. And it's not just for toy stuff. It's a great tool for organizing your software architecture for just generally generating pleasant notation for doing things like documentation in the racket system. There's lots of publications and lots of examples on that. And of course, we've implemented the same. We've used the same machinery to also implement teaching languages and to iterate quickly, improving those languages. So I hope this has motivated you to try out the racket system. Again, please check out the GitHub repository if you want to look at the code or send me email or contact me some other ways. If you're interested in this stuff, and I'm looking forward to your question and a little bit of discussion. We'll see. Thank you. Thank you very much, Mike, for the talk and for introducing us to racket. And to show us how to implement a language there. I already saw there was already quite some discussion in the chat, but we have still some questions for everyone. And we start with a question from someone. And the question is why you use these macros instead of simple functions? Because he thinks you can also use just functions to produce print. Yeah, so that question was asked just after I introduced print, right? So it doesn't refer to all the rest that we did there. And indeed print could be a simple function. But indeed, there's a lot of things that can just be functions that don't need to be macros in racket. But and print is one of them. But then I couldn't have used it to demo a single simple macro, right? All the other macros that I introduced are a lot more complicated. So I figured I'd do a simple one first. Very good. Let's go on. There are all the differences between bindings and operators. It would be, uh, entrepreneurs to quote unquote, it would be something I wouldn't say. It would be great if you showed them more in detail, but that's more common. Yeah. So it is a little bit confusing that there's two kinds of variables, right? And I hope throughout the talk, there's enough of them that you can see what the difference is. The reason for that is in that this is sort of an expert level macro system that we're using there. And the simple macro system that's in racket, the construct for that is called syntax rules, knows only pattern variables. So if you deal only with syntax rules, then it's very simple. But if you want to write more complicated macros like the ones that we did, then you naturally sort of incur both the pattern variables and the regular variables. But there's only these two kinds, right? And you get, you get the hang of it by playing around with it. So then there's a comment by senior on who says all the switching between a hash, a back quote and hash kind of reminds her of existing and entering math mode in Latich. Yeah, that's a great analogy, I think, right? In Latech, when you put a dollar sign, you enter this different syntax, this different world there, and then you can exit that again by writing, you know, backslash text RM or something like that. That's exactly the same idea that you have in racket, right? Is that you enter the syntax construction mode and that you exit it to regular code, and then you can re enter it. So you kind of can pop, pop, pop, push, push, you know, like the like the infinite turtles, I guess. The next question is, if there is, if you have control over the runtime of your language within racket? Yeah, so I see the question goes on a little bit that suggests that memory management might be a subject. So, so racket gives you great control over the runtime because it has one of the most flexible runtimes in existence, it might be the most flexible one in existence, specifically when it comes to control structures. So, so macros is one aspect of the flexibility of racket. Another one is that it has something called the limited control, which allows you to implement exceptions to implement threading, all kinds of things in terms of the primitive constructs of the language. It does not give you control over memory management directly. I mean, you can influence the way that it deals with memory a little bit. It does not give you control whether it's interpreted or compiled, but if you want an interpreted language, you just write an interpreter. So I guess it gets you control over that. But there's a great many examples that come with a racket language and its package ecosystem of different languages. So one example is that there's a Haskell like language, for example, that comes the only difference is that it has parentheses. Or I think the racket based system comes with a language that does functional reactive programming, where each value changes over time, you can see a change live in the rebel. The next question is from the coder, did you implement a whole basic interpreter in racket and how does it compare in speed with a more traditional interpreter? So well, I implemented a bit more than I showed, but I didn't put all the features of AlphaSoft basic and the graphics in there. You notice because it's implemented in macros, though, it is a compiler effectively. So again, it's compiled into a racket code. And I think it would compare quite favorably because racket has a pretty efficient compiler in the backseat there. So it's not apples to oranges, it's kind of apples to 40 year old prunes. And racket obviously doesn't run on an Apple too. But I think the Apple, the AppleSoft and the basic implementation that you're getting there is going to run quite fast, actually. The next question is from Proless. Do you think Racket with its extensive macro system is particularly well suited for building languages? So it is the best system in the world to build languages, right? And so in my commercial work, right, whenever we need to design a DSL, then Racket is at least involved in the prototyping stage. And it could very well be involved in the production stage as well. Specifically, you've noticed that Racket is a Lisp descendant, I guess in many ways. And of course, if you get a more traditional Lisp like Clojure, it also has a macro system. But the macro systems of traditional Lisp are significantly less powerful than the one in Racket. And there's two differences that are relevant. One of them is that the syntax and traditional list often doesn't track source code information. So you don't really get good error messages, which are important if you really have domain specific languages that are exposed to domain experts. And the other one is that it deals with hygiene. We didn't talk about this in the talk because there's lots of other talks and papers on the topic of hygiene. But you know that sometimes when you write a macro, you need to invent new names that are just supposed to be temporary. And doing that requires extreme care in a traditional Lisp system, whereas in Racket, it's just automatic. And so pretty much the subtleties that I've shown, that's a pretty complete set. I don't think I deal with more subtleties on a day-to-day basis. And in a traditional Lisp system, you would also need to deal with hygiene issues, which you don't in Racket. So that gets you sort of one level of power above traditional Lisp in terms of how funky you can get with your DSL.
 Hi there! This is a talk about creating your own programming language, specifically a language  that doesn't suck. And for many of us, the language that does not suck is the first language  that we learned. And for me, this was the language that ran on this computer here, the  Apple 2e, which was a computer that my parents gave to me in 1983. And I just took that picture  today, it still works, and it runs a language called Applesoft Basic. You can still find  documentation on Applesoft Basic to this day. So here's for example, an introductory manual.  And if you look inside that manual, you can see what a basic program looks like. You can  see it consists of a bunch of lines. Every line has a line number, and on every line,  there is a command. So it's none of that methods and classes and functions, nonsense that we  have today. It's just straight up do this, do that, jump to that line, and it was a lot  of fun during the day. So we're going to use the Racket system to implement that. And Racket  is great. It's an entire toolbox that you can use to create your own languages. It comes  with many languages. And it's just a lot of fun. And if you want to play with the code  for this talk yourself, I put everything on GitHub in a repository called MikeSperber  slash RC3. And that also links to a download link for Racket and has all the code for today.  So anyway, I have to warn you, this talk is pretty heavy on code. So that's pretty much  the only thing that I'm going to be doing is I'm going to write code in front of your  eyes. And if that's just too much for you today, I'll understand and I won't be mad  if you go see another talk. So but if you're in for some really serious hacking, then this  might just be the talk for you. So Racket, when it starts up, comes up with a blank screen,  such as this one was quite blank. Each file in Racket starts with a hash lang line that  tells Racket what language that file is written in. And the system actually comes with many,  many languages, one of which also happens to be called Racket. Now, Racket's okay language,  but it also has some annoyances. So we could print something out by calling the display  function. And if we run that program, well, down here, you can see the output. That's  fine. But sometimes we also want to display several things, for example, a little bit  of explanatory text along with a computation. So this year, well, you can see that the default  Racket language is a Lisp dialect, which always has parentheses around compound forms, and  has an operator always in front. So you don't write, you know, one plus one, you write the  sum of one and one. Now we really, well, we hopefully we know what that should print out.  But if we run that, we get an error message. And that's because display only takes a single  argument to print out. And the second argument is supposed to be a port, which tells display  where the output goes, if it goes to a file or the repl or whatever. So really, if we  want, you know, several pieces of output, such as this, then we need to put in several  calls to display. So there you go. Well, that's super annoying, right? And even, you know,  Apple soft basic at the thing where you could just say print, and you could just put in  several things. So you could do this thing that we were trying to do with display in  the first place, right? But of course, you know, run this print, it's just not built  in in the shape that we would like. So we're just going to define it ourselves. And to  do that, we're going to write a macro, which means that the this form with print at the  front is going to get translated into code that racket already knows about. And so we  want to translate this into these two calls to display up here, right? And well, one restriction  that we need to remember is that, you know, a form always translates into a single form.  So when there's two forms here, in order to turn these into a single form, we just put  begin at the front and parentheses around. So this is the translation that we want. I'll  comment that out just so that we have that at our disposal. And in order to write a compile  time function or a macro, we say define syntax, we're going to define a new piece of syntax.  And that piece of syntax always starts with open parent print. And the compile time function  that implements the macro always gets called with an argument, which is exactly the piece  of syntax that the compiler saw. And we're going to call that form. Now, in order to translate  it into this begin form, we need to take it apart a little bit, identify the arguments,  we need to parse it. And for parsing, we will use a racket library called syntax parse.  And I'm going to import that as a library. So it's not a default feature in the core  of racket language. And since we were going to use syntax parse inside a function that  runs at compile time or at syntax expansion time, as we like to say, we're going to say  up here for syntax, so we can use it inside a define syntax form. So we write, you know,  syntax parse here, we want to take apart this form thing. And we will write a pattern that  describes the shape of the form. So we'll start with a very simple version of that pattern.  So it starts with an open parent print, then there's an argument and that argument has  to be an expression, which is denoted by putting colon expression there. And now we want to  translate that into a different piece of syntax. In order to create syntax, what we do is I  put in hash back quote, we want to translate that into display arc for now. So we'll just  do a single argument version for starters. So I'll put that here, I'll run the code.  So there's nothing in the REPL now, but now I can say at least Hello World, very exciting,  and that works. But of course, it will only work with a single argument. We want to use  it with several arguments. In order to do that, we can just say, well, we don't just  want one arc, we want several. And to do that, we put three dots here. Almost looks too  simple to be true. So it just says, well, there can be as many arguments as we want.  And now, as we said above, we need to translate into a begin form. So we want to say like  this, right? And then we want to have as many calls to display as there are arcs. And we  can also do that by putting, you know, three dots here. And Racket will automatically figure  out that that means we should have one call to display for each argument. And make sure  that all the parentheses are closed. And let's try that out. One plus one equals plus like  this. And sure enough, that works. So now we've added a new syntactic form to the Racket  language. Pretty simple as it goes. And we'll use that as a small building block for our  basic machinery that we're going to implement next. So, well, I guess it's a good time to  save our work. Just to make sure that we're not losing anything, I'm just going to call  this file basic dot racket. And there we go. So, okay, well, we got that print thing,  which is similar to the print in basic, which is nice. But I don't know if you've ever used  basic, it looks a basic program really looks like this, right? It says, you know, print  hello world. And then you can see that there's a line number in the beginning, and then you  go something like, yeah, go to 10. So it's always the first program that just print something  repeatedly. So, yeah, well, this is obviously not there yet, right print, whether there's  no line number. There's also still the funny parenthesis and tax. So if you create when  you create a new language in racket, the way to do that is, well, you start with a parenthesis  syntax. And later on top of that, we will implement the actual line based basic syntax  that we might remember from our Apple two days. So, to that end, we'll just start by  saying not, you know, 10 print, what we'll say open parent 10 print, you know, hello,  world. Eventually, we're also going to have to take care of the lowercase print here,  and then say 20, you know, go to 10, something like this. So that's what we're going to start  with. Again, we have to think about how we will translate this into code that racket  recognizes. And, well, you know, this program, it doesn't do anything yet. It just stores,  you know, the print command, for example, it just stores that into into storage of the  of the Apple soft interpreters, and I can then start the program by saying, well, I want  to start the program at line 10, or something like that. So it's not enough to just sort  of translate this into a call to print, we will also have to deal with the line numbers  somehow so that also go to 10, for example, makes sense. So in order to store a piece  of code, we can just give it a name and racket, right? So the idea is that we will translate  it like this, we will say, well, we'll call we'll just define a function, we're just going  to call that line 10. And we will make that one call print. And we will then define a  function called line 20. And well, so go to of course, in racket doesn't exist, but we  can just go to another function by calling it, right? So I'm just going to call line  10 here. And well, you can already see that eventually this this is hardly going to work  as line 10 will never proceed to line 20. So eventually, we will also have to put a  call to line 20 here, so that the code moves in the right direction. But we'll start simple,  we're just going to start with a single line, and then we don't need to call that next function.  How does that work? Well, we again say define syntax, right? And in this case, we're just  going to define a piece of syntax that we're just going to call basic. Again, it takes  an entire form as its argument, and we'll start just with a single command. So we're  going to call syntax powers here, pass it the form and say, well, basic. And the way  basic commands work is there's a line number beginning, which is an integer. And then there  is a command, which is in racket terms, it's just an expression. So, well, what do we need  to do? Again, we need to translate it into another piece of syntax. So, well, we can  see up there, right? We want to translate it into define. And then, well, there's an  open parent, we're going to have to cook up that line 10 name. Also, we'll defer that  problem for a little later. And we will just insert in here the command in the body of  the function. So, okay, so that leaves the problem of computing the name line 10 from  the line. So we want to compute that name by sticking that, you know, line minus in  front. And to that end, well, I want to call a function called format it and format it.  Well, so it takes something that I'll explain in just a moment. It takes a pattern, which  is sort of like printf, except, well, the pattern language is slightly different. So  the tilde a says we should insert something in here and we want to insert the line number.  Now, I'm kind of tempted to put in line number here. Now, I need you to understand one thing  is that, you know, as syntax, a piece of syntax that represents a number is not the same thing  as the number itself. So, for example, if the line number is 10, you know, down here  in the REPL, I can type 10, I get 10 as a result. But if I type this funny syntax thing,  then it says, well, there's a piece of syntax and it contains that number 10 in there. If  we want to stick it into the name, we really need the number 10 in order to extract the  sort of the actual piece of data that is represented by the syntax, I can call a function called  syntax E. So if I call syntax E of, you know, here, hashback quote 10, then it gives me  that 10. So really, you know, that's what that line number is. So I need to call syntax  E here. Moreover, you can see here that I left three dots here, format it also takes  an argument that is called the syntactic context. Now, the macro system in Racket has great  sophistication when it comes to identifiers. If you've seen macro systems in other languages,  such as in C, they're very simple about names. And you can introduce name clashes, very easily  and Racket avoids that. That's not going to be the main subject of this talk. So suffice  it to say that as the context, we want to define the line numbers are at the same level  as the basic form that appeared. So we'll just stick this here. So just a little piece  of magic here that we're just going to have to accept for the moment. So well, this thing,  we want to stick that in here, right in here where the three dots are. And to that end,  I will just give it a name. I'll call that thing name. And I want to stick it in here.  Now you have to understand a very subtle issue here, which is that, so for example, you know,  when I have a pattern variable here that appears in the pattern, I can just stick that in the  syntax. For example, here I can just put in command and Racket will figure out that I  really meant this command here. Unfortunately, well, I mean, there are reasons, but this,  that does not work for this, which is a regular binding. It's not a pattern binding. In order  to tell Racket to, well, you know, stick the result of evaluating that piece of code here,  I will need to put in hash comma name, which says, well, here's an expression, you know,  stick the value of evaluating that expression in here. So that's, I think that's confusing at  first, but you'll get used to it. So remember, when there's something in here in the pattern,  you can just stick it in here. And if you have regular bindings, or if you have a piece of code  that you want to stick in your syntax, you will need to prefix that with a hash comma,  which is also called unquote. So let's see if that works. We'll run that. See here,  I already ran into the first trap here that I just told you about. I called syntax E on the line  number, but the same kind of kind of goes in reverse. So if I can't just use a pattern  variable in regular code, instead of instead, what I need to do is I need to prefix that pattern  variable with hash back quote. Okay. So and then it knows, okay, I need to look among the pattern  variables for this. So if I run this, well, then it says format it, it doesn't know about that,  and that's because format it in racket is in a separate library. So I'm going to need to jump  upwards a little bit and say here that for syntax, I don't just want syntax powers. I also want a  library called racket syntax, and that hopefully contains format it, go back to that code, press  run, and now at least it's silent. And now, well, I could try doing that by saying basic, you know,  10, you know, print Hello world. And doesn't do anything, right, as it should, it's just to  just stash that piece of code. And but if I now type this, it executes that first line of code. So  hooray, we've got a very basic, basic version of basic here. That works. Okay, so now we have a  single command and a basic program. But of course, we want to have several of them. And so up here,  we already know how to do this, or at least how to tell syntax powers about this, we just put three  dots here and says there can now be several lines in a, in a basic form. So, but of course, we need  to translate that differently. Right now, our expansion just has a single definition for a  single line. And now we need to expand that to do several definitions. So to that end, we will  record as a functional programming language, we do a map. So we apply a function to all the line  numbers and commands, and the function always returns a piece of syntax for a single definition.  Let's try that. So I'm going to say lambda here. I'm going to put three dots here, because there's  a subtly that I'll explain in just a moment. And now the question is, what do we map over? Well,  we want to map over all, you know, all the line numbers and all the commands, of course. So if we  have line number here, well, you know, it's not just one line number, right? It's several line  numbers, one for each line in the program. So we always have to use it like this, right, that we  always put the three dots behind it. Otherwise, racket will complain. And moreover, line number  again is a pattern variable that comes from this line in the syntax parse form. So, you know, the  pattern variables, they really only work in this hash bang, this hash back quote form. So that's  what we need to write. And also we do the same thing with the commands. And we're still not done  because, well, you remember, you know, hash back quote returns a syntax object. But what we really  need is a list because we're feeding that into map. So racket has a handy function that does that  called syntax to list. Also, here is syntax to list for the commands. And that allows us to map  over, well, here, the line numbers and the commands. Now, you know, now here's the subtly.  Suddenly, because we now have line number and command be regular lambda parameters, they are no  longer pattern variables. Okay, they're just regular variables. And this, I think, really is the most  subtle aspect of the, of the racket macro system is sort of, you understand this issue, right,  the pattern variables come from up here. And anything, everything else, everything that is not  here in a syntax parse form in the, in the pattern part is not a pattern variable. So in this case,  these are now regular variables. And that means they should not appear in hash back quote anymore.  In this part where the line number is, and also means that a racket is not just going to replace  the pattern variable by the expansion down here with commands. So I've got to push hash comma  here. So if you figure that out, if you figure out that distinction, I think you'll be fine. So  let's try that out. I'll run the program. Thankfully, there are no error messages. Let's try again  that basic program. Oh, it complains again, there's another trap. Well, not a trap. Really, it's  something that we've seen before is map returns a list. And you would see here. So it says your  quote open for N. That means it returned a list. And that means, well, map produced a list,  but racket really expects a syntax object. You know, if you're coming from lists or from closure,  you expect sort of lists to be usable as syntax objects, but racket makes a strict  distinction here. And it uses that distinction to track source code locations to give you good  error messages and also to track hygiene to keep track of lexical binding, which is not the main  focus of the tutorial, but just so you know. So anyway, we need to definitely stick hash back  quote here. And now we have several definitions. And because we have several definitions that we  want to stick in one single form, we use begin just like we did with print before, just going to  indent everything nicely. And of course, now we want to stick the result of evaluating this in  here. So I've got to put hash comma here. Okay. And unfortunately, there's still another subtly. If  you look at the expansion here, you can see that it would, well, hopefully you can see that it would  expand into something like this, right? You have begin, you know, from here. And then you have the  open paren, which is from the list produced by map. And then inside those open paren's are the  definitions of the various lines, like, you know, line 10, line 20, and so on. And you can see that  there's one pair of parentheses too many. So we really don't want the racket macro expander to  stick that list in there. We want it to stick the elements of the list one by one inside the  begin. And to do that, we need to put a magic character here, also sometimes pronounced unspicing.  So and that does exactly what we want. So it's just like hash comma, but it expects a list and it  will stick the elements of the list there effectively removing one pair of parentheses. So  let's try that again. Oops. So here's that basic form. At least it doesn't error out anymore. Let's  see if we can start it. And it says, Hello world. And we could also now extend it, right? And put  another line here. So right now we only know print. So do that. And so we can start line 10. And you  can see that it, well, line 10 runs, but really we expect the program to go on to line 20 after  line 10. Line 20 is there, but we haven't yet linked the lines together. So that's going to be our  next problem. So what we need to do here is, well, after the command, you know, associate with the  line happens, we need to stick in a call to the next line, right? Now, how do we get that call,  right? In order to produce the call for the next line, we need the line number of the next line.  But we don't have it here. We just have the current line number. And to do that, we're just going to  map over an additional list, which is going to be like the original list of line numbers, but we'll  shift it by one. So along with the current line number, there's always the next line number. And  to that end, I'm going to call the list function, CUDR, which just, you know, removes the first  element from the list. And so that gives us that. And now, of course, map would complain because the  lists do not no longer have the same length, right? This list is one shorter. And so we need to  append an element to make up for the last element. And since there is no line after the last line,  I'm just going to pass false, right? This is a racket syntax for false hash f. And we'll need to  make sure that after the last line, we don't try to stick in a call. Okay. So here we are. We need  to, of course, make sure that now that we're mapping over three lists and not just one, I need to add  an additional parameter here, right? And now we could say call next line. And I could just, you  know, stick in a call to format it like this, right? And then pass next line number. But that would  be poor abstraction. And well, it's kind of copy paste code. So we could pull it out into its own  function definition. That's a useful exercise. So I'm going to take that code, copy it one last  time, stick it up here. Now there's two subtleties. Let me get rid of this obsolete comment here that  we have to pay attention to. So I'm going to call that function, you know, make line name. And of  course, we need to pass in the line number, this thing here. But we need to pass in one more thing  because this basic here that you see here, remember that it referred to this basic down here, right?  It used to be here. But that's no longer in scope because we're outside of its parentheses. So we  also need to pass it in as an argument. And we'll call that, it's called the context, used by  Rackett to make sure that the definitions appear in the right place and are referable by their name.  Okay, so we have this great. And then we can take out those two calls here and call make line name  instead. Remember that we now need to call this basic thing that was there before and line number.  And the same thing over here. I'm sorry, it's called make line name, probably noticed. And we'll  do the same thing here, call it make line name, remove all the crud here, let's duplicate it. And  now we have a nice call. Of course, well, we'll get to the fact that it's not a call yet in a  moment. But also, I want you to think about one more aspect of MaxRexpansion. This definition  up here of make line name is just a regular procedure definition. But we want to call it from  the macro expansion process. So essentially at compile time, we're a macro expansion time. But  this is just a regular runtime function. It will not be available until runtime happens. And in  order to tell Rackett, no, this is a function, please make it available to macro expansion. We  need to replace the define by define for syntax. So define it. So it's available during syntax  expansion. Okay, so one more thing. This is of course not a call yet. In order to have a call,  we need to have parentheses to the parentheses around the name of the next line. So I'm going  to put do this and put parentheses around it and then stick this in with hash comma, right. And  that would be the call to the next line. Well, except it isn't quite because remember, next line  number can be false, right. And then we would not have the then then we would not have a function  to call there. So we really need to make a case distinction here. I'm going to say, well, if there  is a next line number, then this is a great function to call. This is the function of the next  line. And if there isn't, well, I'm just going to call void, which is a function that does  absolutely nothing that's built into Rackett. Okay, let's try it out. Let's give it a whirl. Well,  oh, it says next line number, reference to an unbound identifier. That's because I put a type  over here. Let's try that again. Okay, now it at least goes through. Let's see, we had that  program that had two lines in it, doesn't error out. That's great. Call line 10. And, ah, okay,  now it runs both lines. That's wonderful. But we see that we made a small mistake in the  implementation of print, at least in Apple South Basic, when you call print, it prints return a  new line at the end. So why don't as a last act of this particular task, we go up here and stick  up here, they called a new line. That should work. Try that again. And now we get the output as we've  wanted it from the beginning. So now we can have a sort of a basic program that has several lines  where the lines are run consecutively. That's great. So what's next? Well, here we can already  see what might be next, which is the infamous go to command in basic, which just jumps to a  to another place with no memory of where it came from, or what was going on beforehand. So if we  look at, you know, our code here, we can see that go to is going to require special treatment. We  can't just implement it the way that we implement it, that we implement print, because of course,  when there's a go to, there should not be a call to the next line after the go to that would make  no sense because go to just transfers control directly. Now, well, things are getting a little  bit unwieldy at this point in this in this longest function here. So I will put the code that  translates a single command into a separate function, and we'll see how that goes. So here,  again, remember that we need to say define for syntax, so it's not a macro, it's a function that  is called from a macro definition, going to call that function translate command. And well,  so we pass in the, we pass in the actual command. So that's the, that's, if you will, the original  source code from the basic program. And we need to translate that into a racket code. And well,  and the idea is that we use translate command to produce the body of this function here. So,  of course, that means we need to also pass in call next line. So let's see, let's pass that in.  Great. And for now, we're just going to keep that very simple. We're just going to, going to keep  the definition from up here. Now, again, remember that when we produce syntax, we can only produce  a single form at a time. So there are two forms here, but we know we can always combine two forms  into one by sticking a begin in front. So we'll just do the abstraction for now without adding  additional functionality. So instead, we will say, okay, translate command, command, and call next  line and omit this to a way with that, make sure that all the parentheses line up. Yeah,  it's getting, getting to be a lot of parentheses and see if that works. So run that and, you know,  program runs as before. That's great. So we want to give go to special treatment. And in order to  do that, well, we need to inspect the command here and see, well, with, if it's a go to command,  then we give it this treatment. In all other cases, we just translate to the code that we already  have. So I call syntax powers, call pass the command. And now I put here a pattern where it says,  well, if it's a go to command, looks like this, go to, there's a line number and that line number is  an integer, then I'm going to do one thing. And in all other cases, I want to do something else.  And you can see here that, you know, this is now a syntax parse that has several clauses, and it  just tries them in order with the compiler. While it does syntax expansion, first tries the first  one, then the second one, the first one that matches gets to be run. And this thing here is just a  wildcard pattern that matches anything that you want. So that's the fall through clause at the end.  Okay, so in this case, well, we have go to now, here's another subtly, in order, we really want  the word go to to appear there. But remember that syntax parse does pattern matching. So whenever  there's a name here, that name is just going to match whatever is there in the input. It's really  a subtle issue. And we really want the word go to to appear here. And another subtle issue is that  we want that to be consistent with the rest of the binding structure of the program. We'll ignore  that here. But in order to do that, we need to put something like this, a magic clause here at the  beginning that says, well, go to whenever it appears here, means that we really want the word go to  to show up there. So for all of our specially treated commands, we need to put that in, we put  them, you need to put the name of that command into the literals clause up here. Okay, so now,  when there's a go to, we don't, you know, we just want to generate a call to the procedure that's  associated with that line number. And we get the name of that procedure by saying make line name.  Well, and remember, make line name now needs that context argument up here, and we don't have it  here. So we need to add another parameter, call that context up here, we pass basic. And okay,  got that here. And then we can call make line name with context and the line number. Okay, so  got that here. And that's just the name of the procedure, we still need to make a piece of  syntax with a call to it. So we put hash bang back quote, put a pair of parentheses. And in the  middle, we need to put hash bang comma to actually stick that name there. Okay, well, let's see if  it does anything. Oh, error message. Well, again, I fell into the same trap that I always fall into  line number here is a pattern variable. So it really is only valid inside a hash bang back quote.  Well, it is inside a hash bang back quote, but the hash bang comma kind of undoes that it's a, so  we refer to it as a regular variable here, in order to refer to a pattern variable. Again,  we need to put hash bang back quote here. So it must seem confusing at this point. But if you just  do a bunch of stuff with macros, you will soon get the hang of that, I think, and then you'll  recognize the error message right here, it says pattern variable cannot be used outside of a  template. So line number is a pattern variable. Now we can see, oh, that nicely, that occurrence here  of line number nicely refers to that one. Let's see if it runs now. Okay, see if we can do this.  So, well, here's a rule that I forgot about, but fortunately, it's easy to remember because  there's an error message says literal is unbound in phase zero, right? Phase zero has to do with  the phase zero's runtime. Don't have to worry about that too much, but it just means that go to has  no definition that it refers to. And we're not interested in a definition because we're translating  calls to go to into something else anyway. So I'm just going to say put in a dummy definition here  to false. And that just means if there's any reference to that go to, then, you know, syntax  parse will recognize it. Let's push the wrong button there. Let me try again. And, well, at least it  doesn't error out. Let's see if we can run the program. And that's pretty neat. And now, well,  I guess it's time that I put an actual basic program here into the source code so I don't have to  keep retyping it like this. You know, like this, for example, and how are we going to see that it  works? I'm just going to put a line 20 there that says go to 40, make that line 30, and put line 40  which says, you know, the end, right, so that we know the program is done. Let's see if that works.  We could call line 10. And indeed, it says how the world the end and it skips hello again. So go to  works now. All right, what's next? Well, we can just keep adding commands, right? But of course,  some cause more trouble than others. And one that causes a little bit of trouble is assignment, right?  We can have variables in basic might look like, well, in basic it would look like this, right? A  equals 42. And, you know, that would establish a variable called a, and we could then use it, right?  We could say something like this. And I'm going to use a syntax slightly different. So of course,  it has to be parenthesized, the operator has to be in front. So might look like this. And I'm going  to use colon equals to avoid confusion with the with the equals sign without a colon, which is the  equality operator. Okay. So we'll just use the same technique that we used before. So I'm going to  just like go to right needs to be a literal, we want another case to match only when there's  an actual colon equals sign there. Well, here's a variable. And that has to be an identifier.  That's how we talk a tell syntax parts that needs to be an identifier. And then there's a right hand  side, which can be any expression that we want. Okay. So got this here. And now we just need to  generate a piece of code that works with it, right? And in, in racket, variable assignment happens  with an operator that's called set bang. So the exclamation mark is pronounced bang, to denote  that there's an evil side effect happening here. So, and we can just use these two pattern  variables, we can just use the right hand side here. And, okay, well, remember that trick that  we needed here with go to we need it with colon equals to so we need to provide a dummy definition.  And now, well, here's a little problem, right? It says, a is an unbound identifier down here.  And that's because, well, racket distinguishes between binding an identifier. So you're establishing  that the identifier exists and then changing the value in the storage cell associated with  that identifier. Those are two separate things, right? And so we could make that error message  go away by saying, you know, define a to some dummy value. But then of course, that's no longer  basic, right? It's, if we would need that additional definition in basic, we can just introduce  an identifier by doing an assignment to it. So somehow we need to generate the set of variables  that appear in assignments and make sure that we also generate definitions for those identifiers,  dummy definitions at the beginning. So, well, we need to generate those definitions. And for  generating those definitions, we need the set of variables that are in a program. We'll start by  just collecting the variables that are in a single command. And to that end, I'm just going to  define a new function here, call that collect variables. And I'm going to pass in the command.  Okay. And well, not just the command, but we will want to collect a list or rather a set  of those variables, because of course, a variable might occur in several assignments. So we will  use something called an it set. And we'll define that in a minute. So a racket comes with a library  that allows you to manage sets of these identifiers. And we will need to import that library up here.  So call that syntax, that's called syntax it set. And that allows us to deal with those sets.  Okay. And of course, we need to look inside those commands. So we're just going to use  the same syntax part essentially the same syntax parts that we had up here, right? So we have  various literals here with go to, of course, there can't be any variables introduced by a go to,  we're mainly interested in the clause that has to do with assignments. And well, how does that work?  So in that case, what we need to do is we need to say, free it set at and it set and add the variable  here. And anything else, well, we're not going to do anything. I'm just going to call void the  function, the built in function that doesn't do anything. Okay. Okay. And now we need to call  that function collect variables for each command. And we need to do that up here, from the basic  form, right, where we have all the commands at our disposal. So first of all, we need to create  an empty it set that we can add to call it it set. And so we're going to call a function,  it's got a cumbersome name mutable, it's set. Oh, and not just that mutable free it set,  because again, racket makes subtle distinction between identifiers, depending on whether they're  free or bound, don't worry about that. This, this is going to work. And then what we do is, well,  we want to call collect variables on each command and passing in that it set. And but remember,  we don't just have a single command, we have a whole list of them. And so we need to do this. Well,  we take the command dot dot dot, and of course, need to wrap that hash back around it,  convert that into a list, we've done that before. And then we can use a built in function called  for each, which just calls this function on every argument on every element of that list,  right? It's sort of like map, but that returns your list of the result, whereas for each just  doesn't bother with the results and throws them away. So now we have a set of identifiers.  We just need to generate definitions from it. And these definitions need to go here,  right, before the actual code for the basic program. So again, we can use map, right,  to map all over all the variables. We'll figure out in a moment what we're going to do with those.  And well, we'd like to map over it set. Unfortunately, it set is not a list and map  once a list. So we are going to use a function called free, it's set to list to make it a list,  right? And so for each variable, we want to generate a dummy definition. Again, we need to  generate a piece of syntax says define, we stick that variable in, and we'll just define it to  false to make sure that things don't get mixed up with regular basic values. Okay, now remember,  again, I mean, you can see here the other called a map, right, we want to make sure that this gets  spliced into that form here. So again, I call that I use that same funny magic thing here,  hash comma at, which splices all those definitions into that begin form. Okay, well, let's see if  that works. And now it says, oh, that variable is already defined. That's a good sign, because  you can't have several definitions for a single variable. So we put one in ourselves. I'm going  to delete that. And okay. And now we can see that there's a variable here, right? It's bound to  false as it should be. And let's try running the program. Well, not much happens. Why is that,  right? Why does the program not produce anything? Well, we could look at a is now 42. Let's see,  let's go up here. And we can see here that translation of colon equals and you see, you can  see I forgot something. This just sets the variable, but it does not go on with the program as  the rest does. We need to put a begin here and then put in the call to the next line.  Let's see if that works. And now line five. And that actually prints out the value of that variable.  So now we've got variables in our program. That's wonderful. Next up are conditionals in basic.  So how does that work? Well, basic has a statement called the if statement might look like this,  right? We compare a with 42. It's kind of silly. We know what the result is, but just to demonstrate  the feature. And then depending on whether that's true or false, we could whatever said  B to one or set B to two. And we'll just print that out along with the A here, just so we know  what came out. All right, well, we need to translate the if command, of course. So we need  to go up here into our translate command function. We need to add if here to the list of literals.  So we can recognize ifs. And it looks like this. Well, we have some tests. That's an expression.  We have a then branch. That's an expression. And we have an else branch. That's an expression.  And well, we're just going to translate it into brackets if, but we need to make sure that the  then branch and the else branch also get translated. So we use hashback quote. Again,  you know, this is not the same if as here. This is now brackets if we're just going to  leave test as it is. And then we call translate command. And remember, we need we want the  result of translate command stuck in there. So we need to put hash comma in front, pass in some  context. You know, again, you know, we need to pass pattern variable here. So we need to put  hashback quote in front and we'll pass call next line as before. And do the same thing here,  translate command context and call call next line. Okay, well, let's try running it.  That doesn't work, right? We get an error message says B is unbound. Why is that? Well, we're  generating definitions for all the variables that occur. But we but so far, we're only looking at  assignments that occur at the top level, we're not looking inside an if so we need to make sure  that we also extend collect variables in the same way that we extended translate command.  So we also need to add that to the list of literals here and just put the same, we can just  copy the pattern from up here. Copy that here. And well, we just need to call collect variables  recursively on the two branches, right? So we call collect variables on  then with the it set. And we call collect variables with the else on the it set.  Well, let's see if it gets any better. Well, at least it doesn't air out.  Let's show up our program again, we could call it starting at line number five.  And that's not bad. So we now see the it says Princeton number one, which comes from this  assignment here, right? And so B is assigned to one if A is 42, when A is indeed 42, we could also  set it to 41. Try that again, line five. And we see that it now prints a two because the branch  went the other way. I'll just change that back and save. Great. So we'll add one more feature.  That's actually not just one command, but we'll need three commands. And that is basic sub routines.  So what could that look like? For example, in line 1000, we could say we're just going to print  the B variable. And then in line 1010, we just put a return statement. So  Applesoft basic doesn't have functions in the way that we know them, or methods, it just has these  sub routines. And so we could call, we could output B by just saying, you know, whatever line eight  here, and we could say, go sub line 1000 like this. And the idea is then that it will go and print B,  right? And return and go on after that with Hello World and whatnot. We could also call it several  times, of course. Okay. And yeah, we'll just leave it like that, two commands. Okay, so go sub and  return. What should we do? Well, we go up here. And obviously, we need to do something with  translate command, we will just add go sub and return. And these are not built into rackets. So  we need to have go sub false dummy definitions for them to do that. And so how does that work?  Well, go sub is kind of like go to accept the program goes on, right? So while there's go sub  line number, which is an integer. And of course, well, we need to generate a call,  just as we do with go to, right? So going to put on just going to duplicate that from up here.  And the only difference is after that call is done, right? When that finishes, then  after that is done, we need to continue with the next line. Unfortunately, we know how to do that  because we've got the call to the next line sitting up here. So I just stick call next line here.  So that's the only difference between go to and go sub. And of course, we also need to implement  return. And well, what happens with return? With return, nothing happens. Also return is  similar to go to and goes up and that it doesn't go to the next line, it just returns. And when  there's an active go to, it will just go back to the line after that, it will go to that call  to call next line here. Let's see if that works. Okay, so at least we don't get an error message  line five. And well, how do we interpret that output? I kind of forgot what the program was.  Well, here it said goes up 1000, it printed B, and then it went on with the program as before.  Arguably, the program should finish here after line 40 and before line 1000, but I'll  leave that for you as an exercise to implement. So now we have enough basic there to at least  give us a fair idea of how to do this whole thing and maybe how to add the rest. And I  promise that we would also get actual basic syntax and we only have a few minutes left.  So let's get to it. In order to do that, we need to make sure that the functionality that we've  implemented in this file is available in other files. And so we need to add what's called a  provide form. That's like an export or public annotation or something like that. And we just  need to provide the basic form that we implemented, the print command that we implemented, go to,  you know, all those things that we added to implement our basic syntax return. There we go  and save that file and then we're done with that part. Now for the actual basic syntax,  there's no way around writing a proper parser and we don't have nearly enough time to do that.  It's not that hard, but it takes a little bit of time. So I've prepared the code to do that.  It's just called a reader. So here's a file called basic reader dot racket. And if I run that, well,  you can see here, for example, there's a function called parse line takes two arguments, one of  them is called Zerts. That's just context information about what file this code resides in and so on.  I'm just going to pass false here. Later that value will be provided by the racket system.  And the other argument is an input port, which is just a reference to an open file or something  like that. But we can also use a string by just saying open input string here. So if I say, you  know, 10 print a plus one and call that function, you can see that it returns list structure that  corresponds exactly to, you know, the syntax that we, the parenthesis syntax that we implemented  earlier. We just need to hook this up to the rest of the racket system. And for that, we need to  write a little boilerplate function going to call that basic read syntax also takes a Zerts and  an in argument. And that function calls parse program, which does the same thing as parse line,  but for an entire program do this. And we want to embed the result of that in something that  racket will later understand. So do that. And we need to create a module declaration for a racket.  I'm just going to call that module basic. My name doesn't particularly matter. It imports the racket  language. And it also requires so it imports the code that we've just written, which is in the racket  dot arc, KT file. And now here's the result of parsing the program, we still need to put a basic  form around it and splice in the result of parsing the program. So that's that. Now you might have  noticed that the back quote up here and the comma at here doesn't have a hash with it. It doesn't  create syntax structure, it just creates an ordinary racket list structure. We still need to convert  it to a syntax object. And for that, we use a function called datum to syntax and just pass  false here for the context. That's enough. Make sure that all the parentheses line up and then  we're good to go. Well, almost good to go. We still need to export this function and we need to have  a different slightly different name for it. We're just going to call it read syntax on export.  And that's a magic name that racket will later recognize. So I'll save that file and then we are  good to go. So in order to try this out, I've prepared another little example file called  basic demo syntax dot rkt. This is what it looks like. And you can see the hash line line up here  declares that we want to use the reader that we just implemented. Basic reader dot rkt. And what  follows is a very much Apple Soft looking program. And I can run that by just typing line 10,  like we used to do that. And we can see that it pretty much works. So there you have it. I mean  Apple Soft is just about as different from usual racket as it can be. And so if you can do that,  then you can implement just about any language you like with the racket system. And it's not  just for toy stuff. It's a great tool for organizing your software architecture for just generally  generating pleasant notation for doing things like documentation in the racket system. There's  lots of publications and lots of examples on that. And of course, we've implemented the same.  We've used the same machinery to also implement teaching languages and to iterate quickly,  improving those languages. So I hope this has motivated you to try out the racket system.  Again, please check out the GitHub repository if you want to look at the code or send me email  or contact me some other ways. If you're interested in this stuff, and I'm looking forward to your  question and a little bit of discussion. We'll see. Thank you.  Thank you very much, Mike, for the talk and for introducing us to racket.  And to show us how to implement a language there. I already saw there was already quite some  discussion in the chat, but we have still some questions for everyone. And we start with a question  from someone. And the question is why you use these macros instead of simple functions?  Because he thinks you can also use just functions to produce print.  Yeah, so that question was asked just after I introduced print, right? So it doesn't refer  to all the rest that we did there. And indeed print could be a simple function. But indeed,  there's a lot of things that can just be functions that don't need to be macros in racket.  But and print is one of them. But then I couldn't have used it to demo a single simple macro,  right? All the other macros that I introduced are a lot more complicated. So I figured I'd do a  simple one first. Very good. Let's go on. There are all the differences between bindings and  operators. It would be, uh, entrepreneurs to quote unquote, it would be something I wouldn't  say. It would be great if you showed them more in detail, but that's more common.  Yeah. So it is a little bit confusing that there's two kinds of variables, right? And I hope  throughout the talk, there's enough of them that you can see what the difference is. The reason for  that is in that this is sort of an expert level macro system that we're using there. And the simple  macro system that's in racket, the construct for that is called syntax rules, knows only pattern  variables. So if you deal only with syntax rules, then it's very simple. But if you want to write  more complicated macros like the ones that we did, then you naturally sort of incur both the pattern  variables and the regular variables. But there's only these two kinds, right? And you get, you get  the hang of it by playing around with it. So then there's a comment by senior on who says all the  switching between a hash, a back quote and hash kind of reminds her of existing and entering  math mode in Latich. Yeah, that's a great analogy, I think, right? In Latech, when you put a dollar  sign, you enter this different syntax, this different world there, and then you can exit that  again by writing, you know, backslash text RM or something like that. That's exactly the same idea  that you have in racket, right? Is that you enter the syntax construction mode and that you exit  it to regular code, and then you can re enter it. So you kind of can pop, pop, pop, push, push,  you know, like the like the infinite turtles, I guess.  The next question is, if there is, if you have control over the runtime of your language within  racket? Yeah, so I see the question goes on a little bit that suggests that memory management  might be a subject. So, so racket gives you great control over the runtime because it has one of  the most flexible runtimes in existence, it might be the most flexible one in existence,  specifically when it comes to control structures. So, so macros is one aspect of the flexibility  of racket. Another one is that it has something called the limited control, which allows you to  implement exceptions to implement threading, all kinds of things in terms of the primitive  constructs of the language. It does not give you control over memory management directly. I mean,  you can influence the way that it deals with memory a little bit. It does not give you control  whether it's interpreted or compiled, but if you want an interpreted language, you just write an  interpreter. So I guess it gets you control over that. But there's a great many examples that come  with a racket language and its package ecosystem of different languages. So one example is that  there's a Haskell like language, for example, that comes the only difference is that it has  parentheses. Or I think the racket based system comes with a language that does functional  reactive programming, where each value changes over time, you can see a change live in the  rebel. The next question is from the coder, did you implement a whole basic interpreter in  racket and how does it compare in speed with a more traditional interpreter?  So well, I implemented a bit more than I showed, but I didn't put all the features of  AlphaSoft basic and the graphics in there. You notice because it's implemented in macros, though,  it is a compiler effectively. So again, it's compiled into a racket code. And I think it  would compare quite favorably because racket has a pretty efficient compiler  in the backseat there. So it's not apples to oranges, it's kind of apples to 40 year old prunes.  And racket obviously doesn't run on an Apple too. But I think the Apple, the AppleSoft and the  basic implementation that you're getting there is going to run quite fast, actually.  The next question is from Proless. Do you think Racket with its extensive macro system is  particularly well suited for building languages?  So it is the best system in the world to build languages, right? And so in my commercial work,  right, whenever we need to design a DSL, then Racket is at least involved in the prototyping  stage. And it could very well be involved in the production stage as well. Specifically,  you've noticed that Racket is a Lisp descendant, I guess in many ways. And of course, if you get  a more traditional Lisp like Clojure, it also has a macro system. But the macro systems of  traditional Lisp are significantly less powerful than the one in Racket. And there's two differences  that are relevant. One of them is that the syntax and traditional list often doesn't track source  code information. So you don't really get good error messages, which are important if you really  have domain specific languages that are exposed to domain experts. And the other one is that it  deals with hygiene. We didn't talk about this in the talk because there's lots of other talks and  papers on the topic of hygiene. But you know that sometimes when you write a macro, you need to  invent new names that are just supposed to be temporary. And doing that requires extreme care  in a traditional Lisp system, whereas in Racket, it's just automatic. And so pretty much the subtleties  that I've shown, that's a pretty complete set. I don't think I deal with more subtleties on a  day-to-day basis. And in a traditional Lisp system, you would also need to deal with hygiene issues,  which you don't in Racket. So that gets you sort of one level of power above traditional Lisp in  terms of how funky you can get with your DSL.
rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_0_59.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_60_179.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_180_239.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_240_287.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_288_335.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_336_479.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_480_527.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_528_635.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_636_683.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_684_1391.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_1392_1463.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_1464_1523.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_1524_1763.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_1764_3071.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_3072_4955.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_4956_6407.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_6408_9299.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_9300_9419.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_9420_10079.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_10080_10679.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_10680_17447.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_17448_17603.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_17604_22283.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_22284_22355.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_22356_22859.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_22860_24803.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_24804_25631.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_25632_26123.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_26124_26171.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_26172_28523.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_28524_29171.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_29172_29459.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_29460_29519.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_29520_31115.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_31116_31379.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_31380_32855.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_32856_32903.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_32904_33875.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_33876_36755.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_36756_39719.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_39720_40223.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_40224_41375.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_41376_42083.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_42084_42131.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_42132_45683.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_45684_46151.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_46152_46199.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_46200_47135.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_47136_49979.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_49980_53183.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_53184_61151.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_61152_61931.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_61932_64583.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_64584_66491.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_66492_66575.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_66576_72455.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_72456_73055.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_73056_76079.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_76080_76787.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_76788_79991.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_79992_80051.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_80052_81935.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_81936_81983.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_81984_82031.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_82032_82943.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_82944_83279.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_83280_83663.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_83664_83711.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_83712_83843.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_83844_84863.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_84864_85343.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_85344_86411.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_86412_86591.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_86592_86651.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_86652_87071.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_87072_87863.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_87864_88151.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_88152_88739.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_88740_89351.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_89352_89471.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_89472_89531.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_89532_90647.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_90648_91007.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_91008_92543.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_92544_93671.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_93672_94847.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_94848_94979.png rc3-chaoszone-83-eng-deu-All_Programming_Language_Suck_Just_Build_Your_Own_-_Language-Oriented_Progr_94980_96071.png
Hi there! This is a talk about creating your own programming language, specifically a language that doesn't suck. And for many of us, the language that does not suck is the first language that we learned. And for me, this was the language that ran on this computer here, the Apple 2e, which was a computer that my parents gave to me in 1983. And I just took that picture today, it still works, and it runs a language called Applesoft Basic. You can still find documentation on Applesoft Basic to this day. So here's for example, an introductory manual. And if you look inside that manual, you can see what a basic program looks like. You can see it consists of a bunch of lines. Every line has a line number, and on every line, there is a command. So it's none of that methods and classes and functions, nonsense that we have today. It's just straight up do this, do that, jump to that line, and it was a lot of fun during the day. So we're going to use the Racket system to implement that. And Racket is great. It's an entire toolbox that you can use to create your own languages. It comes with many languages. And it's just a lot of fun. And if you want to play with the code for this talk yourself, I put everything on GitHub in a repository called MikeSperber slash RC3. And that also links to a download link for Racket and has all the code for today. So anyway, I have to warn you, this talk is pretty heavy on code. So that's pretty much the only thing that I'm going to be doing is I'm going to write code in front of your eyes. And if that's just too much for you today, I'll understand and I won't be mad if you go see another talk. So but if you're in for some really serious hacking, then this might just be the talk for you. So Racket, when it starts up, comes up with a blank screen, such as this one was quite blank. Each file in Racket starts with a hash lang line that tells Racket what language that file is written in. And the system actually comes with many, many languages, one of which also happens to be called Racket. Now, Racket's okay language, but it also has some annoyances. So we could print something out by calling the display function. And if we run that program, well, down here, you can see the output. That's fine. But sometimes we also want to display several things, for example, a little bit of explanatory text along with a computation. So this year, well, you can see that the default Racket language is a Lisp dialect, which always has parentheses around compound forms, and has an operator always in front. So you don't write, you know, one plus one, you write the sum of one and one. Now we really, well, we hopefully we know what that should print out. But if we run that, we get an error message. And that's because display only takes a single argument to print out. And the second argument is supposed to be a port, which tells display where the output goes, if it goes to a file or the repl or whatever. So really, if we want, you know, several pieces of output, such as this, then we need to put in several calls to display. So there you go. Well, that's super annoying, right? And even, you know, Apple soft basic at the thing where you could just say print, and you could just put in several things. So you could do this thing that we were trying to do with display in the first place, right? But of course, you know, run this print, it's just not built in in the shape that we would like. So we're just going to define it ourselves. And to do that, we're going to write a macro, which means that the this form with print at the front is going to get translated into code that racket already knows about. And so we want to translate this into these two calls to display up here, right? And well, one restriction that we need to remember is that, you know, a form always translates into a single form. So when there's two forms here, in order to turn these into a single form, we just put begin at the front and parentheses around. So this is the translation that we want. I'll comment that out just so that we have that at our disposal. And in order to write a compile time function or a macro, we say define syntax, we're going to define a new piece of syntax. And that piece of syntax always starts with open parent print. And the compile time function that implements the macro always gets called with an argument, which is exactly the piece of syntax that the compiler saw. And we're going to call that form. Now, in order to translate it into this begin form, we need to take it apart a little bit, identify the arguments, we need to parse it. And for parsing, we will use a racket library called syntax parse. And I'm going to import that as a library. So it's not a default feature in the core of racket language. And since we were going to use syntax parse inside a function that runs at compile time or at syntax expansion time, as we like to say, we're going to say up here for syntax, so we can use it inside a define syntax form. So we write, you know, syntax parse here, we want to take apart this form thing. And we will write a pattern that describes the shape of the form. So we'll start with a very simple version of that pattern. So it starts with an open parent print, then there's an argument and that argument has to be an expression, which is denoted by putting colon expression there. And now we want to translate that into a different piece of syntax. In order to create syntax, what we do is I put in hash back quote, we want to translate that into display arc for now. So we'll just do a single argument version for starters. So I'll put that here, I'll run the code. So there's nothing in the REPL now, but now I can say at least Hello World, very exciting, and that works. But of course, it will only work with a single argument. We want to use it with several arguments. In order to do that, we can just say, well, we don't just want one arc, we want several. And to do that, we put three dots here. Almost looks too simple to be true. So it just says, well, there can be as many arguments as we want. And now, as we said above, we need to translate into a begin form. So we want to say like this, right? And then we want to have as many calls to display as there are arcs. And we can also do that by putting, you know, three dots here. And Racket will automatically figure out that that means we should have one call to display for each argument. And make sure that all the parentheses are closed. And let's try that out. One plus one equals plus like this. And sure enough, that works. So now we've added a new syntactic form to the Racket language. Pretty simple as it goes. And we'll use that as a small building block for our basic machinery that we're going to implement next. So, well, I guess it's a good time to save our work. Just to make sure that we're not losing anything, I'm just going to call this file basic dot racket. And there we go. So, okay, well, we got that print thing, which is similar to the print in basic, which is nice. But I don't know if you've ever used basic, it looks a basic program really looks like this, right? It says, you know, print hello world. And then you can see that there's a line number in the beginning, and then you go something like, yeah, go to 10. So it's always the first program that just print something repeatedly. So, yeah, well, this is obviously not there yet, right print, whether there's no line number. There's also still the funny parenthesis and tax. So if you create when you create a new language in racket, the way to do that is, well, you start with a parenthesis syntax. And later on top of that, we will implement the actual line based basic syntax that we might remember from our Apple two days. So, to that end, we'll just start by saying not, you know, 10 print, what we'll say open parent 10 print, you know, hello, world. Eventually, we're also going to have to take care of the lowercase print here, and then say 20, you know, go to 10, something like this. So that's what we're going to start with. Again, we have to think about how we will translate this into code that racket recognizes. And, well, you know, this program, it doesn't do anything yet. It just stores, you know, the print command, for example, it just stores that into into storage of the of the Apple soft interpreters, and I can then start the program by saying, well, I want to start the program at line 10, or something like that. So it's not enough to just sort of translate this into a call to print, we will also have to deal with the line numbers somehow so that also go to 10, for example, makes sense. So in order to store a piece of code, we can just give it a name and racket, right? So the idea is that we will translate it like this, we will say, well, we'll call we'll just define a function, we're just going to call that line 10. And we will make that one call print. And we will then define a function called line 20. And well, so go to of course, in racket doesn't exist, but we can just go to another function by calling it, right? So I'm just going to call line 10 here. And well, you can already see that eventually this this is hardly going to work as line 10 will never proceed to line 20. So eventually, we will also have to put a call to line 20 here, so that the code moves in the right direction. But we'll start simple, we're just going to start with a single line, and then we don't need to call that next function. How does that work? Well, we again say define syntax, right? And in this case, we're just going to define a piece of syntax that we're just going to call basic. Again, it takes an entire form as its argument, and we'll start just with a single command. So we're going to call syntax powers here, pass it the form and say, well, basic. And the way basic commands work is there's a line number beginning, which is an integer. And then there is a command, which is in racket terms, it's just an expression. So, well, what do we need to do? Again, we need to translate it into another piece of syntax. So, well, we can see up there, right? We want to translate it into define. And then, well, there's an open parent, we're going to have to cook up that line 10 name. Also, we'll defer that problem for a little later. And we will just insert in here the command in the body of the function. So, okay, so that leaves the problem of computing the name line 10 from the line. So we want to compute that name by sticking that, you know, line minus in front. And to that end, well, I want to call a function called format it and format it. Well, so it takes something that I'll explain in just a moment. It takes a pattern, which is sort of like printf, except, well, the pattern language is slightly different. So the tilde a says we should insert something in here and we want to insert the line number. Now, I'm kind of tempted to put in line number here. Now, I need you to understand one thing is that, you know, as syntax, a piece of syntax that represents a number is not the same thing as the number itself. So, for example, if the line number is 10, you know, down here in the REPL, I can type 10, I get 10 as a result. But if I type this funny syntax thing, then it says, well, there's a piece of syntax and it contains that number 10 in there. If we want to stick it into the name, we really need the number 10 in order to extract the sort of the actual piece of data that is represented by the syntax, I can call a function called syntax E. So if I call syntax E of, you know, here, hashback quote 10, then it gives me that 10. So really, you know, that's what that line number is. So I need to call syntax E here. Moreover, you can see here that I left three dots here, format it also takes an argument that is called the syntactic context. Now, the macro system in Racket has great sophistication when it comes to identifiers. If you've seen macro systems in other languages, such as in C, they're very simple about names. And you can introduce name clashes, very easily and Racket avoids that. That's not going to be the main subject of this talk. So suffice it to say that as the context, we want to define the line numbers are at the same level as the basic form that appeared. So we'll just stick this here. So just a little piece of magic here that we're just going to have to accept for the moment. So well, this thing, we want to stick that in here, right in here where the three dots are. And to that end, I will just give it a name. I'll call that thing name. And I want to stick it in here. Now you have to understand a very subtle issue here, which is that, so for example, you know, when I have a pattern variable here that appears in the pattern, I can just stick that in the syntax. For example, here I can just put in command and Racket will figure out that I really meant this command here. Unfortunately, well, I mean, there are reasons, but this, that does not work for this, which is a regular binding. It's not a pattern binding. In order to tell Racket to, well, you know, stick the result of evaluating that piece of code here, I will need to put in hash comma name, which says, well, here's an expression, you know, stick the value of evaluating that expression in here. So that's, I think that's confusing at first, but you'll get used to it. So remember, when there's something in here in the pattern, you can just stick it in here. And if you have regular bindings, or if you have a piece of code that you want to stick in your syntax, you will need to prefix that with a hash comma, which is also called unquote. So let's see if that works. We'll run that. See here, I already ran into the first trap here that I just told you about. I called syntax E on the line number, but the same kind of kind of goes in reverse. So if I can't just use a pattern variable in regular code, instead of instead, what I need to do is I need to prefix that pattern variable with hash back quote. Okay. So and then it knows, okay, I need to look among the pattern variables for this. So if I run this, well, then it says format it, it doesn't know about that, and that's because format it in racket is in a separate library. So I'm going to need to jump upwards a little bit and say here that for syntax, I don't just want syntax powers. I also want a library called racket syntax, and that hopefully contains format it, go back to that code, press run, and now at least it's silent. And now, well, I could try doing that by saying basic, you know, 10, you know, print Hello world. And doesn't do anything, right, as it should, it's just to just stash that piece of code. And but if I now type this, it executes that first line of code. So hooray, we've got a very basic, basic version of basic here. That works. Okay, so now we have a single command and a basic program. But of course, we want to have several of them. And so up here, we already know how to do this, or at least how to tell syntax powers about this, we just put three dots here and says there can now be several lines in a, in a basic form. So, but of course, we need to translate that differently. Right now, our expansion just has a single definition for a single line. And now we need to expand that to do several definitions. So to that end, we will record as a functional programming language, we do a map. So we apply a function to all the line numbers and commands, and the function always returns a piece of syntax for a single definition. Let's try that. So I'm going to say lambda here. I'm going to put three dots here, because there's a subtly that I'll explain in just a moment. And now the question is, what do we map over? Well, we want to map over all, you know, all the line numbers and all the commands, of course. So if we have line number here, well, you know, it's not just one line number, right? It's several line numbers, one for each line in the program. So we always have to use it like this, right, that we always put the three dots behind it. Otherwise, racket will complain. And moreover, line number again is a pattern variable that comes from this line in the syntax parse form. So, you know, the pattern variables, they really only work in this hash bang, this hash back quote form. So that's what we need to write. And also we do the same thing with the commands. And we're still not done because, well, you remember, you know, hash back quote returns a syntax object. But what we really need is a list because we're feeding that into map. So racket has a handy function that does that called syntax to list. Also, here is syntax to list for the commands. And that allows us to map over, well, here, the line numbers and the commands. Now, you know, now here's the subtly. Suddenly, because we now have line number and command be regular lambda parameters, they are no longer pattern variables. Okay, they're just regular variables. And this, I think, really is the most subtle aspect of the, of the racket macro system is sort of, you understand this issue, right, the pattern variables come from up here. And anything, everything else, everything that is not here in a syntax parse form in the, in the pattern part is not a pattern variable. So in this case, these are now regular variables. And that means they should not appear in hash back quote anymore. In this part where the line number is, and also means that a racket is not just going to replace the pattern variable by the expansion down here with commands. So I've got to push hash comma here. So if you figure that out, if you figure out that distinction, I think you'll be fine. So let's try that out. I'll run the program. Thankfully, there are no error messages. Let's try again that basic program. Oh, it complains again, there's another trap. Well, not a trap. Really, it's something that we've seen before is map returns a list. And you would see here. So it says your quote open for N. That means it returned a list. And that means, well, map produced a list, but racket really expects a syntax object. You know, if you're coming from lists or from closure, you expect sort of lists to be usable as syntax objects, but racket makes a strict distinction here. And it uses that distinction to track source code locations to give you good error messages and also to track hygiene to keep track of lexical binding, which is not the main focus of the tutorial, but just so you know. So anyway, we need to definitely stick hash back quote here. And now we have several definitions. And because we have several definitions that we want to stick in one single form, we use begin just like we did with print before, just going to indent everything nicely. And of course, now we want to stick the result of evaluating this in here. So I've got to put hash comma here. Okay. And unfortunately, there's still another subtly. If you look at the expansion here, you can see that it would, well, hopefully you can see that it would expand into something like this, right? You have begin, you know, from here. And then you have the open paren, which is from the list produced by map. And then inside those open paren's are the definitions of the various lines, like, you know, line 10, line 20, and so on. And you can see that there's one pair of parentheses too many. So we really don't want the racket macro expander to stick that list in there. We want it to stick the elements of the list one by one inside the begin. And to do that, we need to put a magic character here, also sometimes pronounced unspicing. So and that does exactly what we want. So it's just like hash comma, but it expects a list and it will stick the elements of the list there effectively removing one pair of parentheses. So let's try that again. Oops. So here's that basic form. At least it doesn't error out anymore. Let's see if we can start it. And it says, Hello world. And we could also now extend it, right? And put another line here. So right now we only know print. So do that. And so we can start line 10. And you can see that it, well, line 10 runs, but really we expect the program to go on to line 20 after line 10. Line 20 is there, but we haven't yet linked the lines together. So that's going to be our next problem. So what we need to do here is, well, after the command, you know, associate with the line happens, we need to stick in a call to the next line, right? Now, how do we get that call, right? In order to produce the call for the next line, we need the line number of the next line. But we don't have it here. We just have the current line number. And to do that, we're just going to map over an additional list, which is going to be like the original list of line numbers, but we'll shift it by one. So along with the current line number, there's always the next line number. And to that end, I'm going to call the list function, CUDR, which just, you know, removes the first element from the list. And so that gives us that. And now, of course, map would complain because the lists do not no longer have the same length, right? This list is one shorter. And so we need to append an element to make up for the last element. And since there is no line after the last line, I'm just going to pass false, right? This is a racket syntax for false hash f. And we'll need to make sure that after the last line, we don't try to stick in a call. Okay. So here we are. We need to, of course, make sure that now that we're mapping over three lists and not just one, I need to add an additional parameter here, right? And now we could say call next line. And I could just, you know, stick in a call to format it like this, right? And then pass next line number. But that would be poor abstraction. And well, it's kind of copy paste code. So we could pull it out into its own function definition. That's a useful exercise. So I'm going to take that code, copy it one last time, stick it up here. Now there's two subtleties. Let me get rid of this obsolete comment here that we have to pay attention to. So I'm going to call that function, you know, make line name. And of course, we need to pass in the line number, this thing here. But we need to pass in one more thing because this basic here that you see here, remember that it referred to this basic down here, right? It used to be here. But that's no longer in scope because we're outside of its parentheses. So we also need to pass it in as an argument. And we'll call that, it's called the context, used by Rackett to make sure that the definitions appear in the right place and are referable by their name. Okay, so we have this great. And then we can take out those two calls here and call make line name instead. Remember that we now need to call this basic thing that was there before and line number. And the same thing over here. I'm sorry, it's called make line name, probably noticed. And we'll do the same thing here, call it make line name, remove all the crud here, let's duplicate it. And now we have a nice call. Of course, well, we'll get to the fact that it's not a call yet in a moment. But also, I want you to think about one more aspect of MaxRexpansion. This definition up here of make line name is just a regular procedure definition. But we want to call it from the macro expansion process. So essentially at compile time, we're a macro expansion time. But this is just a regular runtime function. It will not be available until runtime happens. And in order to tell Rackett, no, this is a function, please make it available to macro expansion. We need to replace the define by define for syntax. So define it. So it's available during syntax expansion. Okay, so one more thing. This is of course not a call yet. In order to have a call, we need to have parentheses to the parentheses around the name of the next line. So I'm going to put do this and put parentheses around it and then stick this in with hash comma, right. And that would be the call to the next line. Well, except it isn't quite because remember, next line number can be false, right. And then we would not have the then then we would not have a function to call there. So we really need to make a case distinction here. I'm going to say, well, if there is a next line number, then this is a great function to call. This is the function of the next line. And if there isn't, well, I'm just going to call void, which is a function that does absolutely nothing that's built into Rackett. Okay, let's try it out. Let's give it a whirl. Well, oh, it says next line number, reference to an unbound identifier. That's because I put a type over here. Let's try that again. Okay, now it at least goes through. Let's see, we had that program that had two lines in it, doesn't error out. That's great. Call line 10. And, ah, okay, now it runs both lines. That's wonderful. But we see that we made a small mistake in the implementation of print, at least in Apple South Basic, when you call print, it prints return a new line at the end. So why don't as a last act of this particular task, we go up here and stick up here, they called a new line. That should work. Try that again. And now we get the output as we've wanted it from the beginning. So now we can have a sort of a basic program that has several lines where the lines are run consecutively. That's great. So what's next? Well, here we can already see what might be next, which is the infamous go to command in basic, which just jumps to a to another place with no memory of where it came from, or what was going on beforehand. So if we look at, you know, our code here, we can see that go to is going to require special treatment. We can't just implement it the way that we implement it, that we implement print, because of course, when there's a go to, there should not be a call to the next line after the go to that would make no sense because go to just transfers control directly. Now, well, things are getting a little bit unwieldy at this point in this in this longest function here. So I will put the code that translates a single command into a separate function, and we'll see how that goes. So here, again, remember that we need to say define for syntax, so it's not a macro, it's a function that is called from a macro definition, going to call that function translate command. And well, so we pass in the, we pass in the actual command. So that's the, that's, if you will, the original source code from the basic program. And we need to translate that into a racket code. And well, and the idea is that we use translate command to produce the body of this function here. So, of course, that means we need to also pass in call next line. So let's see, let's pass that in. Great. And for now, we're just going to keep that very simple. We're just going to, going to keep the definition from up here. Now, again, remember that when we produce syntax, we can only produce a single form at a time. So there are two forms here, but we know we can always combine two forms into one by sticking a begin in front. So we'll just do the abstraction for now without adding additional functionality. So instead, we will say, okay, translate command, command, and call next line and omit this to a way with that, make sure that all the parentheses line up. Yeah, it's getting, getting to be a lot of parentheses and see if that works. So run that and, you know, program runs as before. That's great. So we want to give go to special treatment. And in order to do that, well, we need to inspect the command here and see, well, with, if it's a go to command, then we give it this treatment. In all other cases, we just translate to the code that we already have. So I call syntax powers, call pass the command. And now I put here a pattern where it says, well, if it's a go to command, looks like this, go to, there's a line number and that line number is an integer, then I'm going to do one thing. And in all other cases, I want to do something else. And you can see here that, you know, this is now a syntax parse that has several clauses, and it just tries them in order with the compiler. While it does syntax expansion, first tries the first one, then the second one, the first one that matches gets to be run. And this thing here is just a wildcard pattern that matches anything that you want. So that's the fall through clause at the end. Okay, so in this case, well, we have go to now, here's another subtly, in order, we really want the word go to to appear there. But remember that syntax parse does pattern matching. So whenever there's a name here, that name is just going to match whatever is there in the input. It's really a subtle issue. And we really want the word go to to appear here. And another subtle issue is that we want that to be consistent with the rest of the binding structure of the program. We'll ignore that here. But in order to do that, we need to put something like this, a magic clause here at the beginning that says, well, go to whenever it appears here, means that we really want the word go to to show up there. So for all of our specially treated commands, we need to put that in, we put them, you need to put the name of that command into the literals clause up here. Okay, so now, when there's a go to, we don't, you know, we just want to generate a call to the procedure that's associated with that line number. And we get the name of that procedure by saying make line name. Well, and remember, make line name now needs that context argument up here, and we don't have it here. So we need to add another parameter, call that context up here, we pass basic. And okay, got that here. And then we can call make line name with context and the line number. Okay, so got that here. And that's just the name of the procedure, we still need to make a piece of syntax with a call to it. So we put hash bang back quote, put a pair of parentheses. And in the middle, we need to put hash bang comma to actually stick that name there. Okay, well, let's see if it does anything. Oh, error message. Well, again, I fell into the same trap that I always fall into line number here is a pattern variable. So it really is only valid inside a hash bang back quote. Well, it is inside a hash bang back quote, but the hash bang comma kind of undoes that it's a, so we refer to it as a regular variable here, in order to refer to a pattern variable. Again, we need to put hash bang back quote here. So it must seem confusing at this point. But if you just do a bunch of stuff with macros, you will soon get the hang of that, I think, and then you'll recognize the error message right here, it says pattern variable cannot be used outside of a template. So line number is a pattern variable. Now we can see, oh, that nicely, that occurrence here of line number nicely refers to that one. Let's see if it runs now. Okay, see if we can do this. So, well, here's a rule that I forgot about, but fortunately, it's easy to remember because there's an error message says literal is unbound in phase zero, right? Phase zero has to do with the phase zero's runtime. Don't have to worry about that too much, but it just means that go to has no definition that it refers to. And we're not interested in a definition because we're translating calls to go to into something else anyway. So I'm just going to say put in a dummy definition here to false. And that just means if there's any reference to that go to, then, you know, syntax parse will recognize it. Let's push the wrong button there. Let me try again. And, well, at least it doesn't error out. Let's see if we can run the program. And that's pretty neat. And now, well, I guess it's time that I put an actual basic program here into the source code so I don't have to keep retyping it like this. You know, like this, for example, and how are we going to see that it works? I'm just going to put a line 20 there that says go to 40, make that line 30, and put line 40 which says, you know, the end, right, so that we know the program is done. Let's see if that works. We could call line 10. And indeed, it says how the world the end and it skips hello again. So go to works now. All right, what's next? Well, we can just keep adding commands, right? But of course, some cause more trouble than others. And one that causes a little bit of trouble is assignment, right? We can have variables in basic might look like, well, in basic it would look like this, right? A equals 42. And, you know, that would establish a variable called a, and we could then use it, right? We could say something like this. And I'm going to use a syntax slightly different. So of course, it has to be parenthesized, the operator has to be in front. So might look like this. And I'm going to use colon equals to avoid confusion with the with the equals sign without a colon, which is the equality operator. Okay. So we'll just use the same technique that we used before. So I'm going to just like go to right needs to be a literal, we want another case to match only when there's an actual colon equals sign there. Well, here's a variable. And that has to be an identifier. That's how we talk a tell syntax parts that needs to be an identifier. And then there's a right hand side, which can be any expression that we want. Okay. So got this here. And now we just need to generate a piece of code that works with it, right? And in, in racket, variable assignment happens with an operator that's called set bang. So the exclamation mark is pronounced bang, to denote that there's an evil side effect happening here. So, and we can just use these two pattern variables, we can just use the right hand side here. And, okay, well, remember that trick that we needed here with go to we need it with colon equals to so we need to provide a dummy definition. And now, well, here's a little problem, right? It says, a is an unbound identifier down here. And that's because, well, racket distinguishes between binding an identifier. So you're establishing that the identifier exists and then changing the value in the storage cell associated with that identifier. Those are two separate things, right? And so we could make that error message go away by saying, you know, define a to some dummy value. But then of course, that's no longer basic, right? It's, if we would need that additional definition in basic, we can just introduce an identifier by doing an assignment to it. So somehow we need to generate the set of variables that appear in assignments and make sure that we also generate definitions for those identifiers, dummy definitions at the beginning. So, well, we need to generate those definitions. And for generating those definitions, we need the set of variables that are in a program. We'll start by just collecting the variables that are in a single command. And to that end, I'm just going to define a new function here, call that collect variables. And I'm going to pass in the command. Okay. And well, not just the command, but we will want to collect a list or rather a set of those variables, because of course, a variable might occur in several assignments. So we will use something called an it set. And we'll define that in a minute. So a racket comes with a library that allows you to manage sets of these identifiers. And we will need to import that library up here. So call that syntax, that's called syntax it set. And that allows us to deal with those sets. Okay. And of course, we need to look inside those commands. So we're just going to use the same syntax part essentially the same syntax parts that we had up here, right? So we have various literals here with go to, of course, there can't be any variables introduced by a go to, we're mainly interested in the clause that has to do with assignments. And well, how does that work? So in that case, what we need to do is we need to say, free it set at and it set and add the variable here. And anything else, well, we're not going to do anything. I'm just going to call void the function, the built in function that doesn't do anything. Okay. Okay. And now we need to call that function collect variables for each command. And we need to do that up here, from the basic form, right, where we have all the commands at our disposal. So first of all, we need to create an empty it set that we can add to call it it set. And so we're going to call a function, it's got a cumbersome name mutable, it's set. Oh, and not just that mutable free it set, because again, racket makes subtle distinction between identifiers, depending on whether they're free or bound, don't worry about that. This, this is going to work. And then what we do is, well, we want to call collect variables on each command and passing in that it set. And but remember, we don't just have a single command, we have a whole list of them. And so we need to do this. Well, we take the command dot dot dot, and of course, need to wrap that hash back around it, convert that into a list, we've done that before. And then we can use a built in function called for each, which just calls this function on every argument on every element of that list, right? It's sort of like map, but that returns your list of the result, whereas for each just doesn't bother with the results and throws them away. So now we have a set of identifiers. We just need to generate definitions from it. And these definitions need to go here, right, before the actual code for the basic program. So again, we can use map, right, to map all over all the variables. We'll figure out in a moment what we're going to do with those. And well, we'd like to map over it set. Unfortunately, it set is not a list and map once a list. So we are going to use a function called free, it's set to list to make it a list, right? And so for each variable, we want to generate a dummy definition. Again, we need to generate a piece of syntax says define, we stick that variable in, and we'll just define it to false to make sure that things don't get mixed up with regular basic values. Okay, now remember, again, I mean, you can see here the other called a map, right, we want to make sure that this gets spliced into that form here. So again, I call that I use that same funny magic thing here, hash comma at, which splices all those definitions into that begin form. Okay, well, let's see if that works. And now it says, oh, that variable is already defined. That's a good sign, because you can't have several definitions for a single variable. So we put one in ourselves. I'm going to delete that. And okay. And now we can see that there's a variable here, right? It's bound to false as it should be. And let's try running the program. Well, not much happens. Why is that, right? Why does the program not produce anything? Well, we could look at a is now 42. Let's see, let's go up here. And we can see here that translation of colon equals and you see, you can see I forgot something. This just sets the variable, but it does not go on with the program as the rest does. We need to put a begin here and then put in the call to the next line. Let's see if that works. And now line five. And that actually prints out the value of that variable. So now we've got variables in our program. That's wonderful. Next up are conditionals in basic. So how does that work? Well, basic has a statement called the if statement might look like this, right? We compare a with 42. It's kind of silly. We know what the result is, but just to demonstrate the feature. And then depending on whether that's true or false, we could whatever said B to one or set B to two. And we'll just print that out along with the A here, just so we know what came out. All right, well, we need to translate the if command, of course. So we need to go up here into our translate command function. We need to add if here to the list of literals. So we can recognize ifs. And it looks like this. Well, we have some tests. That's an expression. We have a then branch. That's an expression. And we have an else branch. That's an expression. And well, we're just going to translate it into brackets if, but we need to make sure that the then branch and the else branch also get translated. So we use hashback quote. Again, you know, this is not the same if as here. This is now brackets if we're just going to leave test as it is. And then we call translate command. And remember, we need we want the result of translate command stuck in there. So we need to put hash comma in front, pass in some context. You know, again, you know, we need to pass pattern variable here. So we need to put hashback quote in front and we'll pass call next line as before. And do the same thing here, translate command context and call call next line. Okay, well, let's try running it. That doesn't work, right? We get an error message says B is unbound. Why is that? Well, we're generating definitions for all the variables that occur. But we but so far, we're only looking at assignments that occur at the top level, we're not looking inside an if so we need to make sure that we also extend collect variables in the same way that we extended translate command. So we also need to add that to the list of literals here and just put the same, we can just copy the pattern from up here. Copy that here. And well, we just need to call collect variables recursively on the two branches, right? So we call collect variables on then with the it set. And we call collect variables with the else on the it set. Well, let's see if it gets any better. Well, at least it doesn't air out. Let's show up our program again, we could call it starting at line number five. And that's not bad. So we now see the it says Princeton number one, which comes from this assignment here, right? And so B is assigned to one if A is 42, when A is indeed 42, we could also set it to 41. Try that again, line five. And we see that it now prints a two because the branch went the other way. I'll just change that back and save. Great. So we'll add one more feature. That's actually not just one command, but we'll need three commands. And that is basic sub routines. So what could that look like? For example, in line 1000, we could say we're just going to print the B variable. And then in line 1010, we just put a return statement. So Applesoft basic doesn't have functions in the way that we know them, or methods, it just has these sub routines. And so we could call, we could output B by just saying, you know, whatever line eight here, and we could say, go sub line 1000 like this. And the idea is then that it will go and print B, right? And return and go on after that with Hello World and whatnot. We could also call it several times, of course. Okay. And yeah, we'll just leave it like that, two commands. Okay, so go sub and return. What should we do? Well, we go up here. And obviously, we need to do something with translate command, we will just add go sub and return. And these are not built into rackets. So we need to have go sub false dummy definitions for them to do that. And so how does that work? Well, go sub is kind of like go to accept the program goes on, right? So while there's go sub line number, which is an integer. And of course, well, we need to generate a call, just as we do with go to, right? So going to put on just going to duplicate that from up here. And the only difference is after that call is done, right? When that finishes, then after that is done, we need to continue with the next line. Unfortunately, we know how to do that because we've got the call to the next line sitting up here. So I just stick call next line here. So that's the only difference between go to and go sub. And of course, we also need to implement return. And well, what happens with return? With return, nothing happens. Also return is similar to go to and goes up and that it doesn't go to the next line, it just returns. And when there's an active go to, it will just go back to the line after that, it will go to that call to call next line here. Let's see if that works. Okay, so at least we don't get an error message line five. And well, how do we interpret that output? I kind of forgot what the program was. Well, here it said goes up 1000, it printed B, and then it went on with the program as before. Arguably, the program should finish here after line 40 and before line 1000, but I'll leave that for you as an exercise to implement. So now we have enough basic there to at least give us a fair idea of how to do this whole thing and maybe how to add the rest. And I promise that we would also get actual basic syntax and we only have a few minutes left. So let's get to it. In order to do that, we need to make sure that the functionality that we've implemented in this file is available in other files. And so we need to add what's called a provide form. That's like an export or public annotation or something like that. And we just need to provide the basic form that we implemented, the print command that we implemented, go to, you know, all those things that we added to implement our basic syntax return. There we go and save that file and then we're done with that part. Now for the actual basic syntax, there's no way around writing a proper parser and we don't have nearly enough time to do that. It's not that hard, but it takes a little bit of time. So I've prepared the code to do that. It's just called a reader. So here's a file called basic reader dot racket. And if I run that, well, you can see here, for example, there's a function called parse line takes two arguments, one of them is called Zerts. That's just context information about what file this code resides in and so on. I'm just going to pass false here. Later that value will be provided by the racket system. And the other argument is an input port, which is just a reference to an open file or something like that. But we can also use a string by just saying open input string here. So if I say, you know, 10 print a plus one and call that function, you can see that it returns list structure that corresponds exactly to, you know, the syntax that we, the parenthesis syntax that we implemented earlier. We just need to hook this up to the rest of the racket system. And for that, we need to write a little boilerplate function going to call that basic read syntax also takes a Zerts and an in argument. And that function calls parse program, which does the same thing as parse line, but for an entire program do this. And we want to embed the result of that in something that racket will later understand. So do that. And we need to create a module declaration for a racket. I'm just going to call that module basic. My name doesn't particularly matter. It imports the racket language. And it also requires so it imports the code that we've just written, which is in the racket dot arc, KT file. And now here's the result of parsing the program, we still need to put a basic form around it and splice in the result of parsing the program. So that's that. Now you might have noticed that the back quote up here and the comma at here doesn't have a hash with it. It doesn't create syntax structure, it just creates an ordinary racket list structure. We still need to convert it to a syntax object. And for that, we use a function called datum to syntax and just pass false here for the context. That's enough. Make sure that all the parentheses line up and then we're good to go. Well, almost good to go. We still need to export this function and we need to have a different slightly different name for it. We're just going to call it read syntax on export. And that's a magic name that racket will later recognize. So I'll save that file and then we are good to go. So in order to try this out, I've prepared another little example file called basic demo syntax dot rkt. This is what it looks like. And you can see the hash line line up here declares that we want to use the reader that we just implemented. Basic reader dot rkt. And what follows is a very much Apple Soft looking program. And I can run that by just typing line 10, like we used to do that. And we can see that it pretty much works. So there you have it. I mean Apple Soft is just about as different from usual racket as it can be. And so if you can do that, then you can implement just about any language you like with the racket system. And it's not just for toy stuff. It's a great tool for organizing your software architecture for just generally generating pleasant notation for doing things like documentation in the racket system. There's lots of publications and lots of examples on that. And of course, we've implemented the same. We've used the same machinery to also implement teaching languages and to iterate quickly, improving those languages. So I hope this has motivated you to try out the racket system. Again, please check out the GitHub repository if you want to look at the code or send me email or contact me some other ways. If you're interested in this stuff, and I'm looking forward to your question and a little bit of discussion. We'll see. Thank you. Thank you very much, Mike, for the talk and for introducing us to racket. And to show us how to implement a language there. I already saw there was already quite some discussion in the chat, but we have still some questions for everyone. And we start with a question from someone. And the question is why you use these macros instead of simple functions? Because he thinks you can also use just functions to produce print. Yeah, so that question was asked just after I introduced print, right? So it doesn't refer to all the rest that we did there. And indeed print could be a simple function. But indeed, there's a lot of things that can just be functions that don't need to be macros in racket. But and print is one of them. But then I couldn't have used it to demo a single simple macro, right? All the other macros that I introduced are a lot more complicated. So I figured I'd do a simple one first. Very good. Let's go on. There are all the differences between bindings and operators. It would be, uh, entrepreneurs to quote unquote, it would be something I wouldn't say. It would be great if you showed them more in detail, but that's more common. Yeah. So it is a little bit confusing that there's two kinds of variables, right? And I hope throughout the talk, there's enough of them that you can see what the difference is. The reason for that is in that this is sort of an expert level macro system that we're using there. And the simple macro system that's in racket, the construct for that is called syntax rules, knows only pattern variables. So if you deal only with syntax rules, then it's very simple. But if you want to write more complicated macros like the ones that we did, then you naturally sort of incur both the pattern variables and the regular variables. But there's only these two kinds, right? And you get, you get the hang of it by playing around with it. So then there's a comment by senior on who says all the switching between a hash, a back quote and hash kind of reminds her of existing and entering math mode in Latich. Yeah, that's a great analogy, I think, right? In Latech, when you put a dollar sign, you enter this different syntax, this different world there, and then you can exit that again by writing, you know, backslash text RM or something like that. That's exactly the same idea that you have in racket, right? Is that you enter the syntax construction mode and that you exit it to regular code, and then you can re enter it. So you kind of can pop, pop, pop, push, push, you know, like the like the infinite turtles, I guess. The next question is, if there is, if you have control over the runtime of your language within racket? Yeah, so I see the question goes on a little bit that suggests that memory management might be a subject. So, so racket gives you great control over the runtime because it has one of the most flexible runtimes in existence, it might be the most flexible one in existence, specifically when it comes to control structures. So, so macros is one aspect of the flexibility of racket. Another one is that it has something called the limited control, which allows you to implement exceptions to implement threading, all kinds of things in terms of the primitive constructs of the language. It does not give you control over memory management directly. I mean, you can influence the way that it deals with memory a little bit. It does not give you control whether it's interpreted or compiled, but if you want an interpreted language, you just write an interpreter. So I guess it gets you control over that. But there's a great many examples that come with a racket language and its package ecosystem of different languages. So one example is that there's a Haskell like language, for example, that comes the only difference is that it has parentheses. Or I think the racket based system comes with a language that does functional reactive programming, where each value changes over time, you can see a change live in the rebel. The next question is from the coder, did you implement a whole basic interpreter in racket and how does it compare in speed with a more traditional interpreter? So well, I implemented a bit more than I showed, but I didn't put all the features of AlphaSoft basic and the graphics in there. You notice because it's implemented in macros, though, it is a compiler effectively. So again, it's compiled into a racket code. And I think it would compare quite favorably because racket has a pretty efficient compiler in the backseat there. So it's not apples to oranges, it's kind of apples to 40 year old prunes. And racket obviously doesn't run on an Apple too. But I think the Apple, the AppleSoft and the basic implementation that you're getting there is going to run quite fast, actually. The next question is from Proless. Do you think Racket with its extensive macro system is particularly well suited for building languages? So it is the best system in the world to build languages, right? And so in my commercial work, right, whenever we need to design a DSL, then Racket is at least involved in the prototyping stage. And it could very well be involved in the production stage as well. Specifically, you've noticed that Racket is a Lisp descendant, I guess in many ways. And of course, if you get a more traditional Lisp like Clojure, it also has a macro system. But the macro systems of traditional Lisp are significantly less powerful than the one in Racket. And there's two differences that are relevant. One of them is that the syntax and traditional list often doesn't track source code information. So you don't really get good error messages, which are important if you really have domain specific languages that are exposed to domain experts. And the other one is that it deals with hygiene. We didn't talk about this in the talk because there's lots of other talks and papers on the topic of hygiene. But you know that sometimes when you write a macro, you need to invent new names that are just supposed to be temporary. And doing that requires extreme care in a traditional Lisp system, whereas in Racket, it's just automatic. And so pretty much the subtleties that I've shown, that's a pretty complete set. I don't think I deal with more subtleties on a day-to-day basis. And in a traditional Lisp system, you would also need to deal with hygiene issues, which you don't in Racket. So that gets you sort of one level of power above traditional Lisp in terms of how funky you can get with your DSL.
programming language, specifically a language that you can use to create your own languages. It Racket language is a Lisp dialect, which always be an expression, which is denoted by putting argument. And make sure that all the parentheses system in Racket has great sophistication when it to be usable as syntax objects, but racket makes getting, getting to be a lot of parentheses and fortunately, it's easy to remember because the phase zero's runtime. Don't have to worry that work? So in that case, what we need to do is translate the if command, of course. So we need the racket system. And the other argument is an language you like with the racket system. And lot of things that can just be functions that don't need to be macros in racket. But and print we're using there. And the simple macro system language and its package ecosystem of different Racket. And there's two differences that are funky you can get with your DSL.