Clojure: a language for problem solvers
Halloway, Stuart
Clojure is a powerful dynamic language that compiles to many target environments, including the JVM, JavaScript, and the CLR. In this talk, you will learn how to think in Clojure, and why you should want to. Clojure encourages functional style with persistent data structures, a rich library of pure functions, and powerful processing support via the seq and reducer abstractions. Clojure implements a reference model for state, where references represent atomic successions of values, and change is encapsulated by value and reference constructors. This reference model is more substantive and suitable to application development than individual techniques such as Software Transactional Memory (STM) or actors. The most important single principle behind Clojure is simplicity. Clojure's abstractions are simple and orthogonal. A la carte polymorphism, careful support for names and namespaces, the reference succession model, and a wide selection of small, composable protocols make Clojure programming swift, surgical and accurate. Clojure's expressiveness does not mean that you have to compromise on power. It is an explicit design goal of Clojure to provide access to the power of the underlying platform, and for programmers never to have to "drop down" to the platform level for performance-sensitive work.
 Good morning everybody. I'll try that one more time. Good morning everybody. They're all out there on the audio. They just haven't had their coffee yet. My name is Stuart Halloway and I'm going to be talking to you today about closure, which is a programming language for problem solvers. And I want to start from first principles and talk about what I want software to be like. And this is something that we're, you know, working towards gradually. I want software systems to be knowledgeable, right? I want them to have valid and correct information about the world and tons of it. I want them to be powerful. And I want to feel powerful as a programmer using it. You get that high when you do something on the computer faster or easier than you thought you could before. It's just an amazing feeling. I want software to be flexible. How many of you have had the experience as a developer of having a business stakeholder come to you and ask for some feature which sounds really incremental? And you had to say to them, eh, I know that sounds really incremental, but actually given the way the system was architected, it's seeing the impossible. Or it's going to cost doubles. Anybody had that experience? That's a terrible feeling. I don't want that feeling. I want the opposite of that feeling where I make the software 10% more capable with 10% additional effort. And that's very tricky to do. I want my software to be smart. I want programs to anticipate my needs, to be out ahead of me, to have capable analysis. I want them to be effective. I want the decisions made by software to actually enact change in the world. Change in my world, change in the business world. I don't just want idle information, I want action. And finally, and this is really something that has become dominant in the past couple of years, I want software to be pervasive. I and all the users that I write for have an expectation that their software is going to be available how many hours a day. 24, right? Everybody expects 24-7 software. And where is the software going to be available? It used to be software ran in big installations. And then we had our own desktop machines. And then we had laptop machines. And now I have an expectation that I can carry a device in my pocket or on my wrist or in a bracelet that's going to be more powerful than the software that put man on the moon. And if that's not true, I'm going to be pretty disappointed. Now, we try to build, this is, and you know, I would say as a guy who's been in the industry for a while, this feels a little bit like scope creep in terms of what our users want. There was a time when we could have made them happy just having anything at all. But they get more and more demanding. Now, we have to build software in a world where we operate under certain physical laws. One of the physical laws that we operate under is that memory is very expensive. In particular, you probably wouldn't want a provision system with more than, I don't know, 100 kilobytes or so. Because it would be extremely costly to do. Secondly, storage is really expensive. And it's really slow and it's not random access. You have to get bits and bytes off of tape and put them back. So that's a big constraint that we operate under. And machines are really special and precious. Machines are not these anonymous things that you can trivially stand up and down. They're these expensive constructs. They're so special and unique that they're dedicated to tasks. We live in a world of dedicated machines. There's a special database machine somewhere. I know it by name. I call it Gandolf. You know, I get to know it. I swap its hardware out over time. And so, what's wrong with this slide? When was this true? 30 years ago. 30 years ago, right? I think, and, but the problem is that we are working with languages and technologies that are incremental improvements on designs from then. And basically what it comes down to is that the dominant paradigm for writing software in the industry right now is really the greatest hits of the 1970s. And this is a big problem and it's what we're trying to solve with closure. One of the problems that the 1970s give us is an approach to data structures that I'm going to call transients. So if you look at the way we store data, we have to store data transiently because memory is expensive and because storage is expensive, right? Remember, we can only have a couple hundred K of memory and, you know, storage is really pricey. And the effect of having storage be transient is that we have mutable data that we bang away at in place. And so, you see the effect of this on our building systems in a bunch of ways. If I have a data structure which is transient, I can't easily or comfortably share it with you, right? Because somebody might be changing it. It might erode out from under them. Likewise, it's difficult for me to distribute something that's transient because if it's transient, something's happening to it. And we all got to go back to the happening place, right, to find out what may have happened to it. Obviously, concurrent access of transient data structures is a catastrophe. Transient data structures have to be consumed eagerly because if you consume them lazily, by the time you get to them, they may have changed. And they're difficult to cache. Has anybody ever worked on problems of cache invalidation? You can't invalidate something that's persistent, but transient things invalidate all the time because they're changing out from under you. And everything is built this way. The collection APIs in Java are based on transient data structures. The collection APIs in.NET are based on transient data structures. Relational databases are based on transient data structures. And no SQL databases are based on transient data structures. So while I'm excited about no SQL as a movement, it hasn't really addressed what I consider to be the biggest problem yet. And all of this transience comes from a set of design principles that made sense when memory was a million times more expensive than it is now. Imagine how much your world would change if something else got a million times easier to do. Imagine if travel got a million times faster. Right, I live in North Carolina in the United States. It would categorically change the way we planned and organized events. If I could move back to the United States a million times quicker than I will, I'd in fact do tomorrow. We've had that million fold change and very few people have gone back and looked at how we write code and said, hmm, maybe we should do something different. Now, this transient data structure problem is the most important single example of a more general problem, which is complexity. And you know complexity by its symptoms. You have a nonlinear difficulty maintaining systems as they grow. Right, and this could be by any definition of growth. This could be by volume of data. This could be by volume of activity. This could be by size of code base. This could be by features. We've all had that experience that the system gets twice as big and how much more costly does it get to maintain? Even worse, we've had that experience of having the inability to make incremental change to a system. How many of you have some piece of your system that you're terrified to touch? Because you have no idea what's going to happen after you touch it. I mean, our programs are starting to be like the weather. Oh, I see the software is raining today. I have no idea how or why that happened, but I guess we'll just have to live with it. And then of course, in the limiting case, you have large efforts that fail entirely. And so you've probably already heard people throw about statistics about 50% of large IT projects fail, blah, blah, blah. Maybe you've had some experience with that. So we want to attack these problems, and closure is an effort to attack these problems. And it does it in several ways. The thing is that closure uses persistent data structures. I can tell you right now that if you learn nothing else from this talk, if you learn nothing else today, the most important thing you could take back to your job if you don't already use persistent data structures is to start using them. This is a language agnostic observation. You don't have to do the F-sharp pass persistent data structures. So there are places that you can go and get this regardless of what platform you're on. This is where the world is going for all the reasons that I said in the table on the previous slide. Closure is powerful and flexible. I feel like I'm able to get more done, and I feel like once I've gotten something done, I can change it and adapt it to my needs. One of the things that falls out of persistent data structures is functional programming. If I'm going to use persistent data structures, everything's immutable. And so I live in that world of FP. And where that tends to run aground is that we do want our programs to have effect in the world. And a purely functional system would just sit there and get warmer. It wouldn't have any input or output. And so there's a bunch of ways people deal with that problem in the functional languages. One way is to model time actually as functionally as well and use monads. And so there are functional languages that go down that road. Another way is to sort of punt a little bit and say, well, you know, use functional when it's suitable and use something else when it isn't. A third way, which is what closure does, is to say that identity in a system, something that's changeable over time, changes via succession of values. So you have an immutable value, and then you have another immutable value, and then you have another immutable value later. Closure is a Lisp. It's actually the least important of the points on here. Obviously, that hits you in the face when you first look at it, but I made the observation on the panel discussion earlier in the week that someone said, do we have too many functional languages? And I said, I don't think we have anywhere near enough. Just to pick one example familiar to heart, you could build a language based on all these principles except for Lisp, and you would have a language that doesn't exist in the current ecosystem. There's not a choice out there that has closure's characteristics minus being a Lisp. Finally, closure is designed by and for professionals for building professional software. It is not designed for hobbyists or entertainment. It is not designed for newbies. It is not designed for academic use. People do all those things, obviously, but when the rubber meets the road and decisions have to get made in the language, the language is squarely aiming at expert professionals getting shit done. So let's look at the language. Closure is built on top of a data format called the Extensible Data Notation, or EDEN. And when you see EDEN in a moment, you're going to think JSON, and it is similar superficially to JSON, and we would have used JSON except we needed the following three characteristics which JSON does not have. We needed a richer set of built-in data types. JSON has string, which ends up having to get multipurpose for a lot of different things. We wanted a generic mechanism for extensibility, so the ability to add new kinds of data structures without requiring existing serializers or consumers to have to know about them in order to do their work, and language neutral. And we didn't want to get into the object game at all. So I sort of think of my experience with data serialization as a sort of Goldilocks and the Three Bears kind of story where, you know, XML is too hot and JSON is too cold and EDEN is kind of just right. And the sense in which it is just right is that it is richer in its data capabilities than something like JSON, but it does not run off the rails in trying to pursue identities or objects or things like that. And for those of you who have done this stuff for a long time, if you remember things like SOAP Section 5, that's the kind of catastrophe that I'm talking about. So let's take a look at EDEN notation. We have strings, they're double-quoted. In Java or C-sharp, there will be strings. We have characters led by a backslash. There will be characters. We have integers, doubles, booleans, true and false. We have nil. The word nil as opposed to null comes from the Lisp world. We have ratio types, and then we have two types that can represent names. We have symbols, which just look like words, and there's a rich set of legal characters, so, you know, plus sign also could be a symbol. And then we have keywords, which just look like words with a colon stuck in front of them. Most of this lines up fairly well with syntax in C-sharp or Java or JavaScript as far as it goes. Then there are four collection literal types. A list is demarcated by parenthesis, and it suggests something that's going to be seen sequentially. So there's a presumption when something is serialized as a list that when you read it back, you're going to get something that is O of n to traverse it. Vectors, on the other hand, are associative collections by index. So the presumption is if you have something that's in the square brackets, that's something that you could efficiently look up the guy at the end in O of one time. Maps are key value pairs. So here we have the keyword A pointing to the value 100, and the keyword B pointing to the value 90. And sets are collections where you have membership. So here we have a set containing the keyword A and the keyword B. I will mention, and you see it here on the slide, commas are white space. So if you want to, you can go back and add commas into this code. And maybe when you get started, you know, you're initially programming and closure or writing Eden data, you will. And then you realize all the time you can save by never having to type a comma again, and you'll stop doing it, or at least most people do. Now, the funny thing is that I have just taught you the closure programming language. I've taught you the entire syntax because as a list, closure programs are written in data. They're not written in text, and they're consumed as data, not as text. So we take data in Eden, it's actually a superset of Eden, deserialize that into data structures, and that's actually what your program is made of. So here's Hello World. If I interpret Hello World as a piece of data, I see a list containing a symbol, printlin, and a string Hello World. If I interpret it semantically, I say, oh, the first symbol in a list is the name of a function to be invoked. So that is printlin Hello World. Now let's define a function. So here, this is the canonical syntax for defining a function in closure. Def in, define a function, greet the function name, a documentation string, an argument list, in this case a single argument, your name, and then stir, in this case, is taking two arguments and sticking them together and making a string Hello your name. But notice that this has not introduced any new syntax. I could also analyze this purely as data. So this is a list containing a symbol, a symbol, a string, a vector with a symbol in it, and a list with a symbol, a string, and a symbol. It's all made out of data. So it's 9, 14, and 52 seconds. That's how far we are into the talk. You now know closure. So we're just going to take a deep breath for a minute and enjoy the fact that it was so easy to learn the language. So now let's get back to those benefits that I wanted to acquire as a software developer when I'm building systems. I want to be power or knowledgeable. And to be knowledgeable, I actually have to have knowledge. I would submit that C sharp and Java and relational and no SQL systems actually don't have knowledge in them. They have machines in them. And you can go and look at the machine, and then you can come back later and look at the machine and what will have happened. It will have moved. It will have moved into some different position. And persistent data structures are a different approach to that. They are immutable, so they can't change. The only change in the world is by function application. Another way to say this in an object-oriented setting is the only way new stuff happens is via constructors. So as a sort of capsule way to do it. Imagine a world where no objects had setters ever. Everything only had getters, and you only made new things via constructors. That's what we're talking about here. Now, of course, in order to do that, we can't be naive about it. And so we have to have good performance, which means that these persistent data structures actually share structure internally. So I'm not suggesting that if you have a map with 100 items in it, and then you create another map that's exactly like that map, but with one more, we now have 100 more things in memory. Those things can share data together and maintain full fidelity access to old versions. So let's go back and look at the characteristics we get out of making these choices. Sharing transient data structures is difficult. They might change out from under you. Sharing persistent data structures is trivial. If I alias something and 100 people are looking at it, what could possibly go wrong? They can't change. Distribution is not trivial, but it goes from being difficult to easy, particularly when you combine it with something like content addressing. I can take big data structures, and we can share them and refer to them by their hash if we want. And once you know you've got it, you can cache it forever. Concurrent access is also trivial. Everything in Clojure is safe for concurrency and safe for being looked at by multiple observers. The access pattern is your choice. You can consume things in an eager fashion or in a lazy fashion. And obviously, caching is easy because things can just stick around forever. What things look like this today? A lot of functional languages support this, so Clojure, F-Sharp, and on the database side, Datomic, has the entire database as a persistent data structure. Now, having made data this way, we want to have power over our data. In the OO world, we get power over our data by having a proliferation of concrete classes. So I'm going to have the person being and the account being and the order being and the order line item being, that sort of thing. In a functional language, we follow more with Alan Perlis, who said, it's better to have 100 functions that work with one data structure than to have 10 functions that operate on 10 data structures. Or in OO, what do we end up having? Is three or four or five functions on thousands of different data structures. And so every time you sit down to write a program, there's a brand new API that somebody made up. In Clojure, every time you sit down to write a program, there's an existing API, and the API is Vectors, Maps, Lists, and Sets. So let's look at those. We'll start with Vectors. Here's a short program that defines the name V to be a vector containing 42, the keyword rabbit, and a sub-vector 1, 2, 3. I should mention, by the way, here, that everything nests. So all the different data structures you've seen nest inside each other arbitrarily. That's generally not true in programming languages that have a lot more syntax. Go to the top of your C-sharp or Java file and write a for loop around your import statements. And you'll get an idea of the kind of thing that I'm talking about. Because don't nest, there's rules about what you can put where, and you get in trouble, and the compiler will not let you move forward if you don't do that. So now having created an associative data structure, associative data structures act as functions. So instead of saying get from V, the item at the index 1, I just say V1. The first thing in V, and it's the one thing in V, obviously we're zero based, so it's not going to be 42, it'll be rabbit. I can peek at a vector, in which case I will see the thing at the end, 1, 2, 3. I can pop a vector, in which case I will see, and that should be popped off, so that the 1, 2, 3 should be gone. And I can take a sub-vector of a vector, so here give me the vector starting at 1, so 42 falls off the front. All of those things are constant time operations. Now let's take a look at a map. So here's a map with three key value pairs, A1, B2, C3. Again, maps are functions, and so instead of saying get from the map B, you just say map B. Keywords are also functions that look themselves up in collections. So everywhere in an object-oriented language where you'd be saying dot get first name, dot get last name, in an idiomatic closure program you would be saying colon first name in function position, the first slot in that list, and then say that guy's going to be 2. I can get the keys out of a map, so there's ABC, you can also get vowels, that would return 1, 2, 3. I can make new maps, so here's an example of one of those constructors. ASSOCI takes a map and some key value pairs and returns a new map. So here I'm taking the map M, ASSOCIing in D4 and C42. Now C used to be 3, obviously that gets overwritten. Although we should be careful about the use of the word overwritten. What happened to M when I did that? Absolutely nothing. I returned a new object that had those characteristics. M is still as it was, and I can dissociate keys from a map. So dissociate D from M. And then you can get to some more advanced ones. So merge with is a function that takes a function. So this is a higher order function when people talk about functional programming doing higher order functions. This says combine M with A2 and B3, but instead of overwriting values where they collide, combine them using the combining function passed into merge with. So in this case merge with is being the pasta plus sign. So it says oh I see, A used to be 1, but I passed in A2. I combine those, 1 plus 2 is 3. And you can go further with dealing with say nested structures. So let's imagine I have a nested structure. John Doe has a name, but John Doe has an address, and the address is actually nested further. The address has a zip code in it. There's a function called get in that will take a data structure and a list of keys and walk the path of keys down to what you're looking for. So that says walk the path of keys from address to zip, so that's going to return 27705. You can also use this to update data structures. So assoc in is the recursive generalization of assoc. It says take this data structure, drill down into it somewhere, and then make a change at that place. So now I've changed the zip code to 27514. And perhaps even the more interesting one is update in. Update in takes a data structure and a path and a function. And it applies that function to whatever finds in the path. So here I can say go to JDoes, address zip, and increment it. So it would go from 27705 to 27706. In a certain sense this is trivial. In fact, it is trivial. On the flip side, if you built an OO program, there would be a special class that represented the person and a special class that would represent the address. And then it would just be busy work to navigate through it because there wouldn't be a generic access path like this. So let's look at sets. Closure has some extensions for set in the set namespace. So that loads an additional namespace. You don't need this for the literals, but you need it for some of the functions. I'm going to define the colors to be red, green, and blue, and the moods to be happy and blue. I can disjoint from a set. So if I disjoint red from red, green, and blue, I'll get back green and blue. I can take the difference between two sets. So the difference of colors and moods subtracts out the stuff in moods. So I subtract out blue and then up with green and red. I can take the intersection of colors and moods. And I can take the union of colors and moods. Not shown here, but I leave it for you to go play with. How many people feel like they have a grasp of the relational algebra that underpins SQL? Anyone been there, done that? So closure.set the namespace implements the entire relational algebra on Closure's data structures. So you can just go in there and party on and project and join and whatever you want to do. So that's generic data structures. I want to feel powerful. And this is pretty darn easy to summarize. The way I want to feel powerful is I want to have bare metal access to my platform. A lot of people mistakenly divide the world into two kinds of languages. Languages that are at the metal platform languages that are more than highly expressive, friendly, or to the program languages. That's a false dichotomy. I want both, please. And so closure provides direct access to the platform. You can construct things on the platform. Most platforms, it's going to be new class name, some argument. In closure, it's always the verb first. One of the things about closure is all action in the world always comes in the first position of a list. So when you get ready to start metaprogramming, it's a beautiful thing. Everything works the same way. So making a new thing, you see the class name and function position. That says make a new red widget. Static members, math.py on the platform would become math slash pi. Instance methods, rand.nextint would become.nextint on rand. Notice again, perfect consistency across the language. Verb always comes first. Grabbing an instance field would be pixels.data on the platform and would be.data pixels on closure. You're not going to see this one very often because even though I wish they would, idiomatically Java and C-sharp programmers don't encourage you to access the fields of their classes. They tend to make them private. The place where you will see that syntax quite a bit is if you're writing a closure program that's targeting JavaScript. Because in JavaScript, people do that all the time. Yes. So there are three platforms, I think, that are super important. And they are Java, the CLR, and JavaScript. There could be others, but those are the three big important ones. When I say bare metal, I mean access to the platform. I mean specifically, if I'm writing a Java project, I don't want to have to drop to Java to do some high performance thing. Now, I am not talking about dropping underneath garbage collection. So I'm not talking about dropping to J&I or whatever the platform access called in C-sharp, I can't remember. Yeah, PNVoc. I'm not talking about dropping to that level. And I'm not talking about competing with the C guys for their bare metal. So I'm not going there. I am talking about platforms that are garbage collected in virtual machine platforms. So chaining access on the platform, it's x.y.z. Enclosure, it's.. at the beginning. And then that says, that's the recursive generalization of do something, right? So person.getaddress.getzipcode. You don't have to write the. again and again. I will point out that a lot of people, when they first look at Lisp, they make this snide joke on Twitter about punctuation. But if you go back and look through these slides, closure has fewer parentheses than the same code in C-sharp or Java would. And furthermore, it has no commas. And let me tell you, I have been writing code in an Algal syntax language for the last week, and it is murder to have to type commas. I feel for people who have to type commas when they're writing code. You can implement interfaces in closure with a form called reify. So this says reify runnable and implement the run method to print hello. And I have one example just to demonstrate my point about platform power. It's actually a JavaScript example. And you can go and read this article. Some of you may have seen David Nolan. He's here speaking at the conference. He's the lead developer on ClojureScript. And he has been doing benchmarking to verify that ClojureScript can do anything that JavaScript can do, so that there's no need to drop down to JavaScript. And so he put together this demo, which I will attempt to open. So this is a port of Notch's JavaScript performance demo. So this is a purely calculative exercise. This is a processor burning thing. It's doing a Minecraft simulation internally. And David has been verifying that ClojureScript is neck and neck with JavaScript at this. And this is even doing things like icky, mutable, imperative, at the bottom of the rendering level, loop stuff. So I would encourage you to check this out if you're skeptical about claims of platform performance. You could do similar experiments with Clojure running on the CLR or Java. Oh, and you know what? I need to turn that off because it literally will burn one core. So my machine would start to sound like a fan. Its job is to use a core as fast as it can. So I would summarize the Platform Interop story in Clojure as simple, wrapper-free. And this is a big deal. The minute you have wrappers talking to Java or CLR or whatever, then you're no longer in the world of getting the direct stuff. You're now in the world of having a value-added service, which becomes a value-subtracted service when you want to do something different. It's performant and it's conformant. It gives you access to the features of the platform. So I'm feeling pretty knowledgeable and powerful. I want to talk now about feeling flexible. And so a big piece of feeling flexible in a language like Clojure is that the work style is interactive. There's this website. You can go check it out called Tri-Clojure. Tri-Clojure runs a Clojure REPL in the browser. And you can go and just start programming. There's also a website called Himera. Himera is a Clojure script REPL running in the browser. Now, because we have only an hour today, I'm not going to do an extended demo of what it's like developing at the REPL, but I want to talk about it. I had dinner last night with Neil Ford, who will be speaking later today. And I was asking him, he's at ThoughtWorks. I said, how do people develop code in the agile world these days? What's the pattern? How do I move quickly? And he said, I think that it's primarily, it's not necessarily test-driven, but it's keeping the test green. It's having a fast feedback loop and you code and you see the test pass and you code and you see the test pass. That's the people who are working the fastest how they work. I don't work like that. The way I work, I would describe as literate planning. I'm people familiar with Nooth's literate programming, the idea that your program would be a document. There are a bunch of people in the Clojure community who are into that. I'm not going to say that I'm that far, but I do most of my work as a programmer, either in a drawing tool, an OmniGraphil, or in an outlining tool. And inside the outlining tool, I have snippets of code that I send to the REPL for evaluation and gradually build up larger and larger programs. I tend not to write unit tests almost ever, because by the time I've done exploring at the REPL, I've produced a pure function, which I've seen to work on a variety of inputs. Instead, I write generative and simulation tests, which are at a higher level. So generative tests will manufacture inputs of different kinds for functions and then verify assertions about those functions. And then simulation tests, really, I think that B's needs in testing, is having a database of activity simulating real-world activity against your system and running it that way. If you're interested in that, I am giving a talk on simulation testing. I think not in the next slot, but the slot after that this morning. But the point I want to make here is that this interactive style of development is incredibly rapid, and it's quite a bit different from IDE-driven development or test-driven development. In addition to that interactivity and being able to build a system quickly, I want to work in a dynamic language. So Clojure is a dynamically typed language. You notice that we had a bunch of data structures up there, but there weren't any classes or types or any of that stuff. And I want to show you what this feels like by refactoring a piece of common Java code into Clojure. So this method is from a library called the Apache Commons. If you have been in the Java community for more than five minutes, Maven, while it was downloading the Internet onto your machine, certainly downloaded the Apache Commons, because it's used, it's a utility library that does common things that Java doesn't ship with. And so this particular, you can try to read it, this particular function is a validator function that verifies that a string is blank. And the way that a string is blank is, well, if it's null, it's blank, and if it's linked to zero, it's blank. And then the complicated part, if it's got stuff in it, but all of that stuff is white space, then it's blank. So I'm going to take this idiomatic Java code, and I'm going to refactor it to idiomatic Clojure code. So I'm going to remove the type declarations, and this is what people think about when they're coming from a statically typed language to a dynamically typed language. They picture the code that they had with all the types removed, right? And I agree with you that that would be a catastrophe, right? If I stopped here, we would be in a bad place, because basically what I just said is you live in a tar pit with good maps, and now you live in a tar pit with no maps. So we've gotten rid of the maps, now let's get rid of the tar pit part. So I'm going to get rid of the class, because we're just going to make a function, and I'm going to introduce a higher order function. So we don't have a higher order function syntax in Java, I'm just going to make this up. So I'm going to say every ch and stir character is white space, and you can sort of imagine what that does, right? That's an expression that returns true if it's true for every guy in the collection. Now, one of the great things about this higher order functional style of programming is once you've written it that way, all those corner cases around it actually disappear. So now I have is blank of a stir is if every character in stir is white space. And then I'll actually drop down to closure syntax. Now, if you compare this with the strongly statically typed slide at the very beginning, this is the kind of advantage that I'm talking about of writing code in a dynamic language. And you end up with a very, very small number of very short, very general purpose things. And these pieces all have to fit together, because if everything wasn't shorter, that would suck, because I would still want to have the roadmaps that static typing gives me. I wouldn't want to go here. And if I had to make as many things as I would have to make to build an OO system, I'd still want the types. But I make 10 times fewer things on small closure programs, and 100 times fewer things on large closure programs than I would make if I had to write them in OO. I absolutely would. So the question was, would you say that it makes a difference which static language you're using? Yes, I think that the reasons that dynamic is better become much more subtle and nuanced when you compare it to something like ML. And I would be perfectly happy. And this is a point I should have made on the static versus dynamic panel earlier in the week. I think we should all get into functional, and then we can argue about the static and dynamic thing, you know, once we're at the end of the road. Because I consider that static dynamic debate is really kind of tiny compared to we need to be more functional and less imperative. And so I think that closure and F-sharp and Scala and ML and OCaml and all these guys, we should all, we're all on the same side here, right? We are all fighting for truth and justice together. So, intelligent. What makes systems intelligent? One thing that makes systems intelligent is being able to specify, do what I want or do what I mean, not how do you do this. And one aspect of that is being declarative. I'm going to give one example of a declarative DSL enclosure that's called destructuring. And the idea behind destructuring is I want to have a way to bind multiple names. I already showed you how to bind one name, right? DefX or letX, right? But I want to be able to bind multiple names, and I want to work against the abstract structure of what I'm working on. So I don't want this to commit me to concrete types. It's, again, another dynamic argument. And I want this to be available everywhere. So anywhere in the language where I would bind names, I want to be able to do this. And there's going to be two forms of it. I'm going to have a vector form, which is going to do sequential destructuring. And I'm going to have a map form that does associative destructure. This is probably the hardest really early roadblock for people in closure unless it's motivated. So I'm going to show an example whose intention is to motivate it. Let's imagine that I was writing a function to make the Fibonacci numbers. In order to make the Fibonacci numbers, I need to maintain two items of state, right? Because every Fibonacci number is built from what? The previous two. So I have to do something pair-wise and then maybe trim that down to actually see the Fibonacci's. So I've written this function next Fib pair, which takes a pair of numbers, and then it makes the second number, the first number of the new pair, and it makes the sum of the two numbers the second number of the new pair. And the problem with this, and then I iterate over it, by the way, iterate is a higher order function that produces an infinite collection. So iterate says take that seed and then do the recursive generalization of whatever the function is. So take 0, 1, call Nick's Fib pair on that, and then on the result, and then on the result, and then on the result, and then on the result. So that produces a sequence from which I can then extract the Fibonacci numbers. But the thing I don't like about this is that there's a lot of noise for picking apart the pair. And so what closure lets me do is you can add this extra pair of square brackets anywhere where you're binding names, like in a function definition, and that extra pair of square brackets says, I don't want to bind a name to the thing that I was given. I want to destructure it. I want to break into it, and I want to bind A to the first piece of it, and B to the second piece of it. And the thing about these kinds of syntactic improvements is they snowball, right? Because that next Fib pair function is so short and so beautiful now, I'm not going to bother to give it a name, right? I'm going to drop it down to where I'm actually doing the job. And that's the kind of thing that allows you to make categorical improvements in the readability of your code. Because now you can say, now you have control of locality, where I had things that were apart, I've now brought them together. I'm not saying you should bring everything together. I'm not suggesting one giant do all function. But I am saying that having the ability to choose where to bring things together allows you to write expressive code. Associative destructuring works the same way. We bind names by keyword lookup. I'm not going to look at this. Let's get past this. The only other example of a little bit of declarative programming that I want to show you, this is another little DSL that's in closure, is solving the actual problem of Lisp for beginners. The actual problem with Lisp is not parentheses. I already showed you that there are less parentheses in idiomatic Lisp than there are in idiomatic Java or C-sharp. The problem is things look inside out. And I'll show you how that starts to happen. So here I have a map. Jonathan's password is secret. And now I want to associate something new onto it. So notice a social appears outside it and the nickname John appears on the other side. So I'm building up on the left and right of the thing I started with. I'm building out. And then I decide to dissociate off the password because I don't want anybody to see that. And now you start to see this thing that builds up inside out. And most programmers, most humans, don't like to read things inside out. How do they like to read things? Left to right. So Clojure has a macro called thread first that reorders programs from inside out to left to right. There's another macro that reorders things from inside out based on the last argument instead of the first argument. So there's actually two flavors of this. But the point is, I guess there's a couple of points. One is, it's one thing to say I don't like inside out reading, but I want to have a language where I can decide what order I do things in. So in Clojure, I could choose to go left to right. I could choose to go inside out. I could be perverse and implement my own macro that made you read things right to left or top to bottom. Or bottom to top. That seems a little crazy. But the thing about this that's important other than just doing it is that these capabilities are built in Clojure. So this is extending the language itself by making these changes. So these DSLs are what you'd call internal DSLs, right? You can do this right in the language. A second thing that's going to make me feel functional is having a powerful set of persistent data structures. We've already talked about this. But I skipped one on purpose. I said we're going to look at maps, vectors, sets and lists, but we didn't actually. Do you remember which one we skipped? I showed you API for three of the four. A lot of stuff coming really fast. So the one we skipped is lists. I didn't show you any functions for manipulating lists. And the reason that I didn't is that in Clojure, you don't actually think about manipulating lists. You think about sequences as an abstraction over any kind of collection. And so the sequence library, although the syntax of the things that are going to be returned, they're going to print with parentheses around them, so you're going to go, oh, that's lists. They're actually not. Sequence is an abstraction over any collection. So sequence is built at the bottom on first, rest and cons. First returns the first item in a collection. Notice that I'm not using it with a list, though, to my point. First is working here with a vector. Rest returns the rest. And cons sticks something on the front. Take and drop are recursive generalizations. So take2 gives me the first two things. Drop2 gives me everything else after I skip past the first two things. I can apply predicates and higher order functions. So I've got every, not every, not any and some for picking apart collections. And my collections can be lazy or infinite. So you already saw making the Fibonacci's. Here you have the whole numbers. Ink zero makes the whole numbers. Cycle12 returns me an infinite collection of 12, 12, 12. RepeatD returns an infinite collection of Ds. I don't actually have infinite business data, but it is nice to be able to treat things. I mean, where do you think infinite? Actually think streaming through data larger than memory. I have a big data job and I want to stream through it. This is really powerful. And of course, map, filter and reduce. So here we have another API, range, which gives me a range up to a number. So range10 is zero through nine. And then I'm showing you map, filter and reduce here. So filter subtracts out of a sequential thing, stuff that or actually keeps in a sequential thing, things that match a predicate. So keep out of this sequence all the odd things. Map, sometimes called collect or select, depending on your language, says pluck. Call this function on every item in the collection to give me the results and then reduce walks a collection with an accumulator. And so I'm just reducing plus. Now, the big thing here is not that I work with lists this way. The big thing here is that I work with everything this way. So CECs work with collections. They work with directories on the file system. They work with the contents of files. They work with XML data. They work with JSON data. They work with results that's coming out of a database. Just to give a slightly larger example, I downloaded the Rotten Tomatoes API yesterday in the hotel room and I asked myself the question, if you look at the movies that are currently topping the charts, what actors are in more than one? So here's how I'd go about doing that. I'd find the JSON API, I'd download it, I'd parse the JSON, I'd walk the movies, I'd accumulate up all the cast members. I would extract their name because the data structure has in it more than names and I just asked about names. I didn't get the frequencies. So if I've encountered an actor more than once, I want to have a number that says, oh, I encountered that guy twice or ten times or whatever. And then I'll sort by the highest frequency. So there's my pseudocode. It's eight lines. There's the actual code. So go to the box office, you're slurp it down, read the JSON, grab the movies out of it, map cat the abridged cast into a collection, map the name out of that collection, grab the frequencies, and sort by the negative of the second entry in the frequencies table, which is going to sort them from highest to lowest. Notice that I also naturally, as a closure programmer, generalize the question. I'm always going to generalize the question and return an infinite sequence, a potentially infinite sequence, because the consumer can then just take the part that they want. Right? They can just go and grab the little bit that they want. And if you're curious, as of yesterday, nobody was in more than two movies that were in the top 50 from the API, but there were a bunch of people that were in two. Yes? So this is a question that comes up commonly, particularly in a statically typed scenario. So I broke the presentation of the API into two pieces, specifically to try to cover this, and I'll be explicit about it here. All of the collection types have APIs that are about them and return them. So map APIs that return maps, vector APIs that return vectors, set APIs that return sets, and then the sequence abstraction takes you into a different world and says, I want to process this sequentially. So you can do either or. You just have to know which one you're doing. So how do you make a map if you have a vector, for example? So there is an eager map called map V. So map V is, and we don't care about the concrete type. It's not how we roll, right? We care about the semantics. And so map is lazy and map V is eager. But so if you wanted to do, you do want eagerness, though. That's right. And so map V will take you to that data structure. So there are paths to what you want, but they're not the most common and enigmatic thing. But it's not forbidden. You'll want it less once you do this for a while, though. So I also want to feel logical. And I'm not going to go through these examples in any detail, but the point that I want to make is that Closure has been a community that is incredibly friendly to adding logic programming. And so logic programming is not, I'm going to go over and invest in a prologue system, right? It's something that just happens naturally inside of a closure programming. There are at least three fairly well-established libraries for doing logic programming in Closure, Cascalog, CoreLogic, and DatomicDatalog. Cascalog, this looks like a query. This says, walk the people data set, pulling out name and age where age is greater than 40, and then walk the people data set, pulling out name and age where age is greater than 50. As an example of two different queries, this is not super interesting. I just copied it off the website. But the actual thing about this is cool, is that this is backed by Hadoop. So this is an extremely powerful small set of lines of code. This says run a parallel query on data sets that could be spread across tens or hundreds of machines and give me back the results. CoreLogic is, this is the classic monkey needs to move a box to the middle of the room problem implemented in CoreLogic. I would encourage you to go download CoreLogic and take a look at it. The work on CoreLogic has been so appreciated in the logic community that the closure community and the scheme community are starting to have a little love in. And the scheme conference next year is actually going to be co-located with the closure conference. And it's been super exciting to have really high powered academic schemers and particularly reasoned schemers, people who are doing logic programming and scheme, sharing ideas and sharing concepts with the closure community. And then here's Datalog in Datomic. So this is the equivalent of SQL, kind of in our world. You can think of expensive chocolate and related product as views. And when you see variables repeated in a query, that causes a join. The thing I like about Datomic Datalog, and I have a personal interest because I helped build it, is that it is just beautiful to read, to do big jobs compared to SQL, right? There's no joining through tables and getting caught in the structure of the join tables and all that sort of muck that you have to do. And all of this stuff is not leaving the language, right? This is not an alternate world that has to be magically brought into the language. This is just people building libraries in the language. Finally, I want to be effective. And I mean the word effective here in the literal sense. I want a program that has effects. This is where functional programming sometimes runs aground. I mentioned before that one of the things people do in functional programming is they sort of wave their hands at this part of it and say, well, you know, you're in Scala, you can say, well, you use functional where it makes sense, and when you need to have an effect, do you have an effect? You do something different. You can take the high road with types and take a monadic approach to effect. Or you can do what closure does, which is to get away from these in place effects by substituting value transitions. So here's where we are today, right? Every little bit of our program is a machine and putting together programs is like assembling things, all of which are moving. So it's no wonder that it's hard to do, right? Composing programs is really difficult because everything is a bunch of moving parts that you're trying to stick together. And this made sense when memory was at 1970s prices. In closure, new memories always use new places. Memory is not that cheap anymore, and we can let garbage collection forget whichever old memories we don't care about. So change is encapsulated by constructors, references refer to a point in time value, and references only see a succession of values. So here we have three values, v1, v2, and v3. We have constructor functions that know how to make values, that maybe take an old value and give you a new one. And then we have atomic succession from one value to another. And then a view of this across time, once upon a time the New York Yankees had this roster, and then somebody got fired and they had that roster, and then two people got hired and they had that roster. Those values are protected by this reference, and observers who are outside that can perceive the identity and remember it and record it, and they can use it in a concurrently safe way. And this is the big one, observers do not have to coordinate. So observers of references or values don't have to coordinate with each other. And people look at this for the first time and go, aren't these just variables? No. They're not just variables. They represent understood semantics of transitions from value to value. If you've ever had to write a lock, then you're doing something different than what we're talking about here. These things do not have to be protected. Here's what it looks like from the API level. I have a reference constructor. In this case, I'm using one called Adam. That makes a reference around the value zero. And then I'm going to take a pure function to do something to the contents of that and some args. And then I have an atomic succession function. So swap says, send a function inside this box, apply the function inside the box, give me back the result. So what will be inside of the Adam after this? It was zero, now it will be 10. And here's mapping that back to the picture. So I have a reference. That reference points to an initial value. I apply a function. We have an atomic succession of the reference, and then it's at a new value. I can do the same thing with, this model is completely general. I can do the same thing with bigger structures. So now the Adam has in it different data. It's actually the equivalent of a Java bean or something. I'm creating a person, and I'm swapping, associating in a different name for that person. I can also have varying semantics. So now instead of having Adam, I have a different kind of ref called a promise, and a different kind of succession function called deliver. So there's not a single semantic for concurrency. There are five or six or seven, depending on how you count them, and they are all safe. This model also scales all the way up to an entire database. So here, I have, this is not closure anymore. This is datomic, which is built in closure. And I'm connecting to a database and transacting against it. Transact is a succession function, just like all of those other succession functions. It's acid, and you start with a value of the database, you get back a new value of the database. In fact, the return value from a transaction is a future. So you can decide whether you want to wait on the return value or not. And inside that future, you have the old value of the database and the new value of the database. It is this atomic succession model writ large. Now, the obvious question, and this came up on the panel, really, you give me back the entire value of the database? What happens if the database is big? What happens if the database is ginormous? And the answer is that these databases are values. So I don't have to give you back, I can evaluate the database lazy. I don't have to give you back the entire database. What can I give you? I can give you the promise that I still know how to get all the rest of it. And then I can fetch the other pieces as needed. So you can grab these slides after. There's a breakdown reference of the various other reference types, but we're not going to look at them today. The last thing I want is for my software to be pervasive. I don't want to have a private island runtime that I have to go and convince managers to install. I want my systems to run on the platforms that people use. And I think those platforms for language like Clojure are the JVM, JavaScript, and.NET. Clojure was originally written for the JVM, that is the most mature implementation, that's where the most work has gone forth. Clojure CLR came next, but Clojure Script has already passed it probably, if not in maturity, at least in widespread use. And the problem, unfortunately, is, and you know this as well as I do, most of you in this room, that languages on the CLR languish until and unless they're blessed by Visual Studio. So I think that Clojure is backwatered on the CLR because there's a strong commitment to using those tools, and the tools are not available. And I would say it would be great if some of you would take it out for a spin and help make Clojure on the CLR more mature and take it in the direction of use that it has seen on Java. So in conclusion, everybody in the industry almost builds things out of transient data structures that don't compose. This includes Java, C-sharp, relational databases, and no SQL. Clojure solves this with persistence and succession of values. Persistence gives us systems that are composable because the parts don't move around while you're not looking. Value succession gives us a way to actually model state that makes sense in that world. There are other choices. You can go down the Monad Road, for example, if you want. These better semantics allow us to write more powerful and more flexible programs, and those semantics are available on real-world platforms where you want to build and run software. I've included at the end of the slides links to a bunch of references. One of these references, and you can easily Google for this. If you Google for Stuart Halloway presentations, I actually have a wiki that's got slides from all the presentations that I do, and I have left, I believe, four minutes for questions. Yes? You mentioned using an outside tool to use the application of the application. Do you have to use a real-world? Yes, so the question is, what is my outlining tool? Yes, I use org mode and snippets. I totally get that that's not where the world is going to go, and I don't want to be an Emacs user. I'm just forced there by its compelling features. But I understand that, you know, telling someone to use Emacs, if you're not already an Emacs user, did you just tell me to F myself? Right? Because it's a hard learning curve. There's IDE support for closure in all of the mainstream Java IDEs, and it has characteristics that are more familiar and friendly to developers that don't matter very much to me, and what it's missing is general text editing capability, speed, and in particular, a structural editing mode called par edit. So if you've never worked in a language that had a structural editing mode, I would encourage you to do a Google search for par edit, P-A-R-E-D-I-T. That's the big reason that most closure users stick with Emacs once they get hooked on it. Other questions? Those of you who are going to be watching this on camera, I'm in a brightly sunlit room, shading my eyes so I can see people. Yes? So the question is, is it possible, given that the data structures evolve by creating new immutable data structures, can you get back to the old ones? The answer is you could if you wanted to, and it depends. So closure does not, by default, keep the last ten versions of a ref and let you go back to them, but the ref semantics, the model of refs, you could write a ref that had those semantics if it made sense to do in your program. Also, obviously, if you hold on to a value, if you go back and look at my picture of the observers, some of them are looking at old value, as long as you're still looking at it, you're still looking at it. Now, on the durable side of persistent data structures, when you go to Datomic, that works a little bit differently because in Datomic, everything is time-stamped and nothing is ever thrown away in that sense. So in Datomic, you can actually put a filter on the database that shears off data from some time range. So you can say dot as of a month ago, teleport your database back in time a month and query against it there. That's the kind, and it's a great example of the kind of thing that's theoretically made possible by this. Not all the reference types deliver that capability, but it's possible. One last question. Okay. Right, so how does this data structure copying thing work efficiently? So if you imagine a linked list, if I have a linked list with four guys on it and you stick a guy on the front, I don't know you did that, and that's called shared structure. This guy's talking to that guy and nobody cares. The data structures in Datomic are all listed in sideways. They're all trees, and they have the same property. So you have an associative data structure over here that's represented as a tree. You make a change to one thing. You get a new tree that's pointing willy-nilly all into the old tree to reach all the old stuff except the one. So the cost you pay is what's called a path copy, and in closure, the paths go five deep. So it's actually, and they are, it's log 32 in instead of constant time, but given that it only goes five deep, it falls out in the wash. In Datomic, on the durable side, those trees are three deep. So they're incredibly wide branching trees, so you don't pay a big penalty for that. That is slower. Those data structure updates are slower than a purely mutable data structure. So on the flip side, you get the speed back on reads, right, because you don't have to coordinate, you don't have to lock. And so if you live in a world where there will be more multi-core processors in the future, closure will become better and betterer. And if you live in a world where multi-core processors are going to be defunct, then the stuff I'm talking about is ridiculous and you should ignore me. So my name is Stuart Halloway. Make sure you remember to fill out eval cards. I have discount coupons for this O'Reilly closure training video and closure stickers. If anybody didn't get those, come and see me after the talk. Thank you very much.
 Good morning everybody.  I'll try that one more time.  Good morning everybody.  They're all out there on the audio.  They just haven't had their coffee yet.  My name is Stuart Halloway and I'm going to be talking to you today about closure,  which is a programming language for problem solvers.  And I want to start from first principles and talk about what I want software to be like.  And this is something that we're, you know, working towards gradually.  I want software systems to be knowledgeable, right?  I want them to have valid and correct information about the world and tons of it.  I want them to be powerful.  And I want to feel powerful as a programmer using it.  You get that high when you do something on the computer faster or easier than you thought you could before.  It's just an amazing feeling.  I want software to be flexible.  How many of you have had the experience as a developer of having a business stakeholder come to you  and ask for some feature which sounds really incremental?  And you had to say to them, eh, I know that sounds really incremental,  but actually given the way the system was architected, it's seeing the impossible.  Or it's going to cost doubles.  Anybody had that experience?  That's a terrible feeling.  I don't want that feeling.  I want the opposite of that feeling where I make the software 10% more capable with 10% additional effort.  And that's very tricky to do.  I want my software to be smart.  I want programs to anticipate my needs, to be out ahead of me, to have capable analysis.  I want them to be effective.  I want the decisions made by software to actually enact change in the world.  Change in my world, change in the business world.  I don't just want idle information, I want action.  And finally, and this is really something that has become dominant in the past couple of years,  I want software to be pervasive.  I and all the users that I write for have an expectation that their software is going to be available how many hours a day.  24, right?  Everybody expects 24-7 software.  And where is the software going to be available?  It used to be software ran in big installations.  And then we had our own desktop machines.  And then we had laptop machines.  And now I have an expectation that I can carry a device in my pocket or on my wrist or in a bracelet  that's going to be more powerful than the software that put man on the moon.  And if that's not true, I'm going to be pretty disappointed.  Now, we try to build, this is, and you know, I would say as a guy who's been in the industry for a while,  this feels a little bit like scope creep in terms of what our users want.  There was a time when we could have made them happy just having anything at all.  But they get more and more demanding.  Now, we have to build software in a world where we operate under certain physical laws.  One of the physical laws that we operate under is that memory is very expensive.  In particular, you probably wouldn't want a provision system with more than, I don't know, 100 kilobytes or so.  Because it would be extremely costly to do.  Secondly, storage is really expensive.  And it's really slow and it's not random access.  You have to get bits and bytes off of tape and put them back.  So that's a big constraint that we operate under.  And machines are really special and precious.  Machines are not these anonymous things that you can trivially stand up and down.  They're these expensive constructs.  They're so special and unique that they're dedicated to tasks.  We live in a world of dedicated machines.  There's a special database machine somewhere.  I know it by name.  I call it Gandolf.  You know, I get to know it.  I swap its hardware out over time.  And so, what's wrong with this slide?  When was this true?  30 years ago.  30 years ago, right?  I think, and, but the problem is that we are working with languages and technologies that are incremental improvements on designs from then.  And basically what it comes down to is that the dominant paradigm for writing software in the industry right now is really the greatest hits of the 1970s.  And this is a big problem and it's what we're trying to solve with closure.  One of the problems that the 1970s give us is an approach to data structures that I'm going to call transients.  So if you look at the way we store data, we have to store data transiently because memory is expensive and because storage is expensive, right?  Remember, we can only have a couple hundred K of memory and, you know, storage is really pricey.  And the effect of having storage be transient is that we have mutable data that we bang away at in place.  And so, you see the effect of this on our building systems in a bunch of ways.  If I have a data structure which is transient, I can't easily or comfortably share it with you, right?  Because somebody might be changing it.  It might erode out from under them.  Likewise, it's difficult for me to distribute something that's transient because if it's transient, something's happening to it.  And we all got to go back to the happening place, right, to find out what may have happened to it.  Obviously, concurrent access of transient data structures is a catastrophe.  Transient data structures have to be consumed eagerly because if you consume them lazily, by the time you get to them, they may have changed.  And they're difficult to cache.  Has anybody ever worked on problems of cache invalidation?  You can't invalidate something that's persistent, but transient things invalidate all the time because they're changing out from under you.  And everything is built this way.  The collection APIs in Java are based on transient data structures.  The collection APIs in.NET are based on transient data structures.  Relational databases are based on transient data structures.  And no SQL databases are based on transient data structures.  So while I'm excited about no SQL as a movement, it hasn't really addressed what I consider to be the biggest problem yet.  And all of this transience comes from a set of design principles that made sense when memory was a million times more expensive than it is now.  Imagine how much your world would change if something else got a million times easier to do.  Imagine if travel got a million times faster.  Right, I live in North Carolina in the United States.  It would categorically change the way we planned and organized events.  If I could move back to the United States a million times quicker than I will, I'd in fact do tomorrow.  We've had that million fold change and very few people have gone back and looked at how we write code and said,  hmm, maybe we should do something different.  Now, this transient data structure problem is the most important single example of a more general problem, which is complexity.  And you know complexity by its symptoms.  You have a nonlinear difficulty maintaining systems as they grow.  Right, and this could be by any definition of growth.  This could be by volume of data.  This could be by volume of activity.  This could be by size of code base.  This could be by features.  We've all had that experience that the system gets twice as big and how much more costly does it get to maintain?  Even worse, we've had that experience of having the inability to make incremental change to a system.  How many of you have some piece of your system that you're terrified to touch?  Because you have no idea what's going to happen after you touch it.  I mean, our programs are starting to be like the weather.  Oh, I see the software is raining today.  I have no idea how or why that happened, but I guess we'll just have to live with it.  And then of course, in the limiting case, you have large efforts that fail entirely.  And so you've probably already heard people throw about statistics about 50% of large IT projects fail, blah, blah, blah.  Maybe you've had some experience with that.  So we want to attack these problems, and closure is an effort to attack these problems.  And it does it in several ways.  The thing is that closure uses persistent data structures.  I can tell you right now that if you learn nothing else from this talk, if you learn nothing else today,  the most important thing you could take back to your job if you don't already use persistent data structures is to start using them.  This is a language agnostic observation.  You don't have to do the F-sharp pass persistent data structures.  So there are places that you can go and get this regardless of what platform you're on.  This is where the world is going for all the reasons that I said in the table on the previous slide.  Closure is powerful and flexible.  I feel like I'm able to get more done, and I feel like once I've gotten something done, I can change it and adapt it to my needs.  One of the things that falls out of persistent data structures is functional programming.  If I'm going to use persistent data structures, everything's immutable.  And so I live in that world of FP.  And where that tends to run aground is that we do want our programs to have effect in the world.  And a purely functional system would just sit there and get warmer.  It wouldn't have any input or output.  And so there's a bunch of ways people deal with that problem in the functional languages.  One way is to model time actually as functionally as well and use monads.  And so there are functional languages that go down that road.  Another way is to sort of punt a little bit and say, well, you know, use functional when it's suitable and use something else when it isn't.  A third way, which is what closure does, is to say that identity in a system, something that's changeable over time,  changes via succession of values.  So you have an immutable value, and then you have another immutable value, and then you have another immutable value later.  Closure is a Lisp.  It's actually the least important of the points on here.  Obviously, that hits you in the face when you first look at it, but I made the observation on the panel discussion earlier in the week  that someone said, do we have too many functional languages?  And I said, I don't think we have anywhere near enough.  Just to pick one example familiar to heart, you could build a language based on all these principles except for Lisp,  and you would have a language that doesn't exist in the current ecosystem.  There's not a choice out there that has closure's characteristics minus being a Lisp.  Finally, closure is designed by and for professionals for building professional software.  It is not designed for hobbyists or entertainment.  It is not designed for newbies.  It is not designed for academic use.  People do all those things, obviously, but when the rubber meets the road and decisions have to get made in the language,  the language is squarely aiming at expert professionals getting shit done.  So let's look at the language.  Closure is built on top of a data format called the Extensible Data Notation, or EDEN.  And when you see EDEN in a moment, you're going to think JSON,  and it is similar superficially to JSON, and we would have used JSON except we needed the following three characteristics which JSON does not have.  We needed a richer set of built-in data types.  JSON has string, which ends up having to get multipurpose for a lot of different things.  We wanted a generic mechanism for extensibility, so the ability to add new kinds of data structures  without requiring existing serializers or consumers to have to know about them in order to do their work,  and language neutral.  And we didn't want to get into the object game at all.  So I sort of think of my experience with data serialization as a sort of Goldilocks and the Three Bears kind of story where, you know,  XML is too hot and JSON is too cold and EDEN is kind of just right.  And the sense in which it is just right is that it is richer in its data capabilities than something like JSON,  but it does not run off the rails in trying to pursue identities or objects or things like that.  And for those of you who have done this stuff for a long time, if you remember things like SOAP Section 5,  that's the kind of catastrophe that I'm talking about.  So let's take a look at EDEN notation.  We have strings, they're double-quoted.  In Java or C-sharp, there will be strings.  We have characters led by a backslash.  There will be characters.  We have integers, doubles, booleans, true and false.  We have nil.  The word nil as opposed to null comes from the Lisp world.  We have ratio types, and then we have two types that can represent names.  We have symbols, which just look like words, and there's a rich set of legal characters,  so, you know, plus sign also could be a symbol.  And then we have keywords, which just look like words with a colon stuck in front of them.  Most of this lines up fairly well with syntax in C-sharp or Java or JavaScript as far as it goes.  Then there are four collection literal types.  A list is demarcated by parenthesis, and it suggests something that's going to be seen sequentially.  So there's a presumption when something is serialized as a list that when you read it back,  you're going to get something that is O of n to traverse it.  Vectors, on the other hand, are associative collections by index.  So the presumption is if you have something that's in the square brackets,  that's something that you could efficiently look up the guy at the end in O of one time.  Maps are key value pairs.  So here we have the keyword A pointing to the value 100, and the keyword B pointing to the value 90.  And sets are collections where you have membership.  So here we have a set containing the keyword A and the keyword B.  I will mention, and you see it here on the slide, commas are white space.  So if you want to, you can go back and add commas into this code.  And maybe when you get started, you know, you're initially programming and closure or writing Eden data, you will.  And then you realize all the time you can save by never having to type a comma again,  and you'll stop doing it, or at least most people do.  Now, the funny thing is that I have just taught you the closure programming language.  I've taught you the entire syntax because as a list, closure programs are written in data.  They're not written in text, and they're consumed as data, not as text.  So we take data in Eden, it's actually a superset of Eden,  deserialize that into data structures, and that's actually what your program is made of.  So here's Hello World.  If I interpret Hello World as a piece of data, I see a list containing a symbol,  printlin, and a string Hello World.  If I interpret it semantically, I say, oh, the first symbol in a list is the name of a function to be invoked.  So that is printlin Hello World.  Now let's define a function.  So here, this is the canonical syntax for defining a function in closure.  Def in, define a function, greet the function name, a documentation string,  an argument list, in this case a single argument, your name,  and then stir, in this case, is taking two arguments and sticking them together and making a string Hello your name.  But notice that this has not introduced any new syntax.  I could also analyze this purely as data.  So this is a list containing a symbol, a symbol, a string, a vector with a symbol in it,  and a list with a symbol, a string, and a symbol.  It's all made out of data.  So it's 9, 14, and 52 seconds.  That's how far we are into the talk.  You now know closure.  So we're just going to take a deep breath for a minute and enjoy the fact that it was so easy to learn the language.  So now let's get back to those benefits that I wanted to acquire as a software developer when I'm building systems.  I want to be power or knowledgeable.  And to be knowledgeable, I actually have to have knowledge.  I would submit that C sharp and Java and relational and no SQL systems actually don't have knowledge in them.  They have machines in them.  And you can go and look at the machine, and then you can come back later and look at the machine and what will have happened.  It will have moved.  It will have moved into some different position.  And persistent data structures are a different approach to that.  They are immutable, so they can't change.  The only change in the world is by function application.  Another way to say this in an object-oriented setting is the only way new stuff happens is via constructors.  So as a sort of capsule way to do it.  Imagine a world where no objects had setters ever.  Everything only had getters, and you only made new things via constructors.  That's what we're talking about here.  Now, of course, in order to do that, we can't be naive about it.  And so we have to have good performance, which means that these persistent data structures actually share structure internally.  So I'm not suggesting that if you have a map with 100 items in it, and then you create another map that's exactly like that map,  but with one more, we now have 100 more things in memory.  Those things can share data together and maintain full fidelity access to old versions.  So let's go back and look at the characteristics we get out of making these choices.  Sharing transient data structures is difficult.  They might change out from under you.  Sharing persistent data structures is trivial.  If I alias something and 100 people are looking at it, what could possibly go wrong?  They can't change.  Distribution is not trivial, but it goes from being difficult to easy, particularly when you combine it with something like content addressing.  I can take big data structures, and we can share them and refer to them by their hash if we want.  And once you know you've got it, you can cache it forever.  Concurrent access is also trivial.  Everything in Clojure is safe for concurrency and safe for being looked at by multiple observers.  The access pattern is your choice.  You can consume things in an eager fashion or in a lazy fashion.  And obviously, caching is easy because things can just stick around forever.  What things look like this today?  A lot of functional languages support this, so Clojure, F-Sharp, and on the database side, Datomic,  has the entire database as a persistent data structure.  Now, having made data this way, we want to have power over our data.  In the OO world, we get power over our data by having a proliferation of concrete classes.  So I'm going to have the person being and the account being and the order being and the order line item being, that sort of thing.  In a functional language, we follow more with Alan Perlis, who said,  it's better to have 100 functions that work with one data structure than to have 10 functions that operate on 10 data structures.  Or in OO, what do we end up having?  Is three or four or five functions on thousands of different data structures.  And so every time you sit down to write a program, there's a brand new API that somebody made up.  In Clojure, every time you sit down to write a program, there's an existing API, and the API is Vectors, Maps, Lists, and Sets.  So let's look at those.  We'll start with Vectors.  Here's a short program that defines the name V to be a vector containing 42, the keyword rabbit, and a sub-vector 1, 2, 3.  I should mention, by the way, here, that everything nests.  So all the different data structures you've seen nest inside each other arbitrarily.  That's generally not true in programming languages that have a lot more syntax.  Go to the top of your C-sharp or Java file and write a for loop around your import statements.  And you'll get an idea of the kind of thing that I'm talking about.  Because don't nest, there's rules about what you can put where, and you get in trouble, and the compiler will not let you move forward if you don't do that.  So now having created an associative data structure, associative data structures act as functions.  So instead of saying get from V, the item at the index 1, I just say V1.  The first thing in V, and it's the one thing in V, obviously we're zero based, so it's not going to be 42, it'll be rabbit.  I can peek at a vector, in which case I will see the thing at the end, 1, 2, 3.  I can pop a vector, in which case I will see, and that should be popped off, so that the 1, 2, 3 should be gone.  And I can take a sub-vector of a vector, so here give me the vector starting at 1, so 42 falls off the front.  All of those things are constant time operations.  Now let's take a look at a map.  So here's a map with three key value pairs, A1, B2, C3.  Again, maps are functions, and so instead of saying get from the map B, you just say map B.  Keywords are also functions that look themselves up in collections.  So everywhere in an object-oriented language where you'd be saying dot get first name, dot get last name,  in an idiomatic closure program you would be saying colon first name in function position,  the first slot in that list, and then say that guy's going to be 2.  I can get the keys out of a map, so there's ABC, you can also get vowels, that would return 1, 2, 3.  I can make new maps, so here's an example of one of those constructors.  ASSOCI takes a map and some key value pairs and returns a new map.  So here I'm taking the map M, ASSOCIing in D4 and C42.  Now C used to be 3, obviously that gets overwritten.  Although we should be careful about the use of the word overwritten.  What happened to M when I did that? Absolutely nothing.  I returned a new object that had those characteristics.  M is still as it was, and I can dissociate keys from a map.  So dissociate D from M.  And then you can get to some more advanced ones.  So merge with is a function that takes a function.  So this is a higher order function when people talk about functional programming doing higher order functions.  This says combine M with A2 and B3, but instead of overwriting values where they collide,  combine them using the combining function passed into merge with.  So in this case merge with is being the pasta plus sign.  So it says oh I see, A used to be 1, but I passed in A2.  I combine those, 1 plus 2 is 3.  And you can go further with dealing with say nested structures.  So let's imagine I have a nested structure.  John Doe has a name, but John Doe has an address,  and the address is actually nested further.  The address has a zip code in it.  There's a function called get in that will take a data structure and a list of keys  and walk the path of keys down to what you're looking for.  So that says walk the path of keys from address to zip, so that's going to return 27705.  You can also use this to update data structures.  So assoc in is the recursive generalization of assoc.  It says take this data structure, drill down into it somewhere,  and then make a change at that place.  So now I've changed the zip code to 27514.  And perhaps even the more interesting one is update in.  Update in takes a data structure and a path and a function.  And it applies that function to whatever finds in the path.  So here I can say go to JDoes, address zip, and increment it.  So it would go from 27705 to 27706.  In a certain sense this is trivial. In fact, it is trivial.  On the flip side, if you built an OO program, there would be a special class  that represented the person and a special class that would represent the address.  And then it would just be busy work to navigate through it  because there wouldn't be a generic access path like this.  So let's look at sets.  Closure has some extensions for set in the set namespace.  So that loads an additional namespace. You don't need this for the literals,  but you need it for some of the functions.  I'm going to define the colors to be red, green, and blue,  and the moods to be happy and blue.  I can disjoint from a set.  So if I disjoint red from red, green, and blue, I'll get back green and blue.  I can take the difference between two sets.  So the difference of colors and moods subtracts out the stuff in moods.  So I subtract out blue and then up with green and red.  I can take the intersection of colors and moods.  And I can take the union of colors and moods.  Not shown here, but I leave it for you to go play with.  How many people feel like they have a grasp of the relational algebra  that underpins SQL?  Anyone been there, done that?  So closure.set the namespace implements the entire relational algebra  on Closure's data structures.  So you can just go in there and party on and project and join and whatever you want to do.  So that's generic data structures.  I want to feel powerful.  And this is pretty darn easy to summarize.  The way I want to feel powerful is I want to have bare metal access to my platform.  A lot of people mistakenly divide the world into two kinds of languages.  Languages that are at the metal platform languages  that are more than highly expressive, friendly, or to the program languages.  That's a false dichotomy.  I want both, please.  And so closure provides direct access to the platform.  You can construct things on the platform.  Most platforms, it's going to be new class name, some argument.  In closure, it's always the verb first.  One of the things about closure is all action in the world  always comes in the first position of a list.  So when you get ready to start metaprogramming, it's a beautiful thing.  Everything works the same way.  So making a new thing, you see the class name and function position.  That says make a new red widget.  Static members, math.py on the platform would become math slash pi.  Instance methods, rand.nextint would become.nextint on rand.  Notice again, perfect consistency across the language.  Verb always comes first.  Grabbing an instance field would be pixels.data on the platform  and would be.data pixels on closure.  You're not going to see this one very often  because even though I wish they would,  idiomatically Java and C-sharp programmers don't encourage you to access  the fields of their classes.  They tend to make them private.  The place where you will see that syntax quite a bit  is if you're writing a closure program that's targeting JavaScript.  Because in JavaScript, people do that all the time.  Yes.  So there are three platforms, I think, that are super important.  And they are Java, the CLR, and JavaScript.  There could be others, but those are the three big important ones.  When I say bare metal, I mean access to the platform.  I mean specifically, if I'm writing a Java project,  I don't want to have to drop to Java to do some high performance thing.  Now, I am not talking about dropping underneath garbage collection.  So I'm not talking about dropping to J&I or whatever the platform access  called in C-sharp, I can't remember.  Yeah, PNVoc.  I'm not talking about dropping to that level.  And I'm not talking about competing with the C guys for their bare metal.  So I'm not going there.  I am talking about platforms that are garbage collected in virtual machine platforms.  So chaining access on the platform, it's x.y.z.  Enclosure, it's.. at the beginning.  And then that says, that's the recursive generalization of do something, right?  So person.getaddress.getzipcode.  You don't have to write the. again and again.  I will point out that a lot of people, when they first look at Lisp,  they make this snide joke on Twitter about punctuation.  But if you go back and look through these slides,  closure has fewer parentheses than the same code in C-sharp or Java would.  And furthermore, it has no commas.  And let me tell you, I have been writing code in an Algal syntax language for the last week,  and it is murder to have to type commas.  I feel for people who have to type commas when they're writing code.  You can implement interfaces in closure with a form called reify.  So this says reify runnable and implement the run method to print hello.  And I have one example just to demonstrate my point about platform power.  It's actually a JavaScript example.  And you can go and read this article.  Some of you may have seen David Nolan.  He's here speaking at the conference.  He's the lead developer on ClojureScript.  And he has been doing benchmarking to verify that ClojureScript can do anything that JavaScript can do,  so that there's no need to drop down to JavaScript.  And so he put together this demo, which I will attempt to open.  So this is a port of Notch's JavaScript performance demo.  So this is a purely calculative exercise.  This is a processor burning thing.  It's doing a Minecraft simulation internally.  And David has been verifying that ClojureScript is neck and neck with JavaScript at this.  And this is even doing things like icky, mutable, imperative,  at the bottom of the rendering level, loop stuff.  So I would encourage you to check this out if you're skeptical about claims of platform performance.  You could do similar experiments with Clojure running on the CLR or Java.  Oh, and you know what? I need to turn that off because it literally will burn one core.  So my machine would start to sound like a fan.  Its job is to use a core as fast as it can.  So I would summarize the Platform Interop story in Clojure as simple, wrapper-free.  And this is a big deal.  The minute you have wrappers talking to Java or CLR or whatever,  then you're no longer in the world of getting the direct stuff.  You're now in the world of having a value-added service,  which becomes a value-subtracted service when you want to do something different.  It's performant and it's conformant.  It gives you access to the features of the platform.  So I'm feeling pretty knowledgeable and powerful.  I want to talk now about feeling flexible.  And so a big piece of feeling flexible in a language like Clojure  is that the work style is interactive.  There's this website. You can go check it out called Tri-Clojure.  Tri-Clojure runs a Clojure REPL in the browser.  And you can go and just start programming.  There's also a website called Himera.  Himera is a Clojure script REPL running in the browser.  Now, because we have only an hour today,  I'm not going to do an extended demo of what it's like developing at the REPL,  but I want to talk about it.  I had dinner last night with Neil Ford, who will be speaking later today.  And I was asking him, he's at ThoughtWorks.  I said, how do people develop code in the agile world these days?  What's the pattern? How do I move quickly?  And he said, I think that it's primarily, it's not necessarily test-driven,  but it's keeping the test green.  It's having a fast feedback loop and you code and you see the test pass  and you code and you see the test pass.  That's the people who are working the fastest how they work.  I don't work like that.  The way I work, I would describe as literate planning.  I'm people familiar with Nooth's literate programming,  the idea that your program would be a document.  There are a bunch of people in the Clojure community who are into that.  I'm not going to say that I'm that far,  but I do most of my work as a programmer,  either in a drawing tool, an OmniGraphil, or in an outlining tool.  And inside the outlining tool, I have snippets of code  that I send to the REPL for evaluation and gradually build up larger and larger programs.  I tend not to write unit tests almost ever,  because by the time I've done exploring at the REPL,  I've produced a pure function, which I've seen to work on a variety of inputs.  Instead, I write generative and simulation tests, which are at a higher level.  So generative tests will manufacture inputs of different kinds for functions  and then verify assertions about those functions.  And then simulation tests, really, I think that B's needs in testing,  is having a database of activity simulating real-world activity against your system  and running it that way.  If you're interested in that, I am giving a talk on simulation testing.  I think not in the next slot, but the slot after that this morning.  But the point I want to make here is that this interactive style of development  is incredibly rapid, and it's quite a bit different from IDE-driven development  or test-driven development.  In addition to that interactivity and being able to build a system quickly,  I want to work in a dynamic language.  So Clojure is a dynamically typed language.  You notice that we had a bunch of data structures up there,  but there weren't any classes or types or any of that stuff.  And I want to show you what this feels like by refactoring a piece of common Java code into Clojure.  So this method is from a library called the Apache Commons.  If you have been in the Java community for more than five minutes,  Maven, while it was downloading the Internet onto your machine,  certainly downloaded the Apache Commons, because it's used,  it's a utility library that does common things that Java doesn't ship with.  And so this particular, you can try to read it,  this particular function is a validator function that verifies that a string is blank.  And the way that a string is blank is, well, if it's null, it's blank,  and if it's linked to zero, it's blank.  And then the complicated part, if it's got stuff in it,  but all of that stuff is white space, then it's blank.  So I'm going to take this idiomatic Java code,  and I'm going to refactor it to idiomatic Clojure code.  So I'm going to remove the type declarations,  and this is what people think about when they're coming from a statically typed language  to a dynamically typed language.  They picture the code that they had with all the types removed, right?  And I agree with you that that would be a catastrophe, right?  If I stopped here, we would be in a bad place,  because basically what I just said is you live in a tar pit with good maps,  and now you live in a tar pit with no maps.  So we've gotten rid of the maps, now let's get rid of the tar pit part.  So I'm going to get rid of the class, because we're just going to make a function,  and I'm going to introduce a higher order function.  So we don't have a higher order function syntax in Java, I'm just going to make this up.  So I'm going to say every ch and stir character is white space,  and you can sort of imagine what that does, right?  That's an expression that returns true if it's true for every guy in the collection.  Now, one of the great things about this higher order functional style of programming  is once you've written it that way, all those corner cases around it actually disappear.  So now I have is blank of a stir is if every character in stir is white space.  And then I'll actually drop down to closure syntax.  Now, if you compare this with the strongly statically typed slide at the very beginning,  this is the kind of advantage that I'm talking about of writing code in a dynamic language.  And you end up with a very, very small number of very short, very general purpose things.  And these pieces all have to fit together, because if everything wasn't shorter,  that would suck, because I would still want to have the roadmaps that static typing gives me.  I wouldn't want to go here.  And if I had to make as many things as I would have to make to build an OO system,  I'd still want the types.  But I make 10 times fewer things on small closure programs,  and 100 times fewer things on large closure programs than I would make if I had to write them in OO.  I absolutely would.  So the question was, would you say that it makes a difference which static language you're using?  Yes, I think that the reasons that dynamic is better become much more subtle and nuanced  when you compare it to something like ML.  And I would be perfectly happy.  And this is a point I should have made on the static versus dynamic panel earlier in the week.  I think we should all get into functional, and then we can argue about the static and dynamic thing,  you know, once we're at the end of the road.  Because I consider that static dynamic debate is really kind of tiny compared to we need to be more functional  and less imperative.  And so I think that closure and F-sharp and Scala and ML and OCaml and all these guys,  we should all, we're all on the same side here, right?  We are all fighting for truth and justice together.  So, intelligent.  What makes systems intelligent?  One thing that makes systems intelligent is being able to specify, do what I want or do what I mean,  not how do you do this.  And one aspect of that is being declarative.  I'm going to give one example of a declarative DSL enclosure that's called destructuring.  And the idea behind destructuring is I want to have a way to bind multiple names.  I already showed you how to bind one name, right? DefX or letX, right?  But I want to be able to bind multiple names, and I want to work against the abstract structure of what I'm working on.  So I don't want this to commit me to concrete types.  It's, again, another dynamic argument.  And I want this to be available everywhere.  So anywhere in the language where I would bind names, I want to be able to do this.  And there's going to be two forms of it.  I'm going to have a vector form, which is going to do sequential destructuring.  And I'm going to have a map form that does associative destructure.  This is probably the hardest really early roadblock for people in closure unless it's motivated.  So I'm going to show an example whose intention is to motivate it.  Let's imagine that I was writing a function to make the Fibonacci numbers.  In order to make the Fibonacci numbers, I need to maintain two items of state, right?  Because every Fibonacci number is built from what?  The previous two.  So I have to do something pair-wise and then maybe trim that down to actually see the Fibonacci's.  So I've written this function next Fib pair, which takes a pair of numbers,  and then it makes the second number, the first number of the new pair,  and it makes the sum of the two numbers the second number of the new pair.  And the problem with this, and then I iterate over it,  by the way, iterate is a higher order function that produces an infinite collection.  So iterate says take that seed and then do the recursive generalization of whatever the function is.  So take 0, 1, call Nick's Fib pair on that, and then on the result, and then on the result, and then on the result,  and then on the result.  So that produces a sequence from which I can then extract the Fibonacci numbers.  But the thing I don't like about this is that there's a lot of noise for picking apart the pair.  And so what closure lets me do is you can add this extra pair of square brackets anywhere where you're binding names,  like in a function definition, and that extra pair of square brackets says,  I don't want to bind a name to the thing that I was given.  I want to destructure it.  I want to break into it, and I want to bind A to the first piece of it, and B to the second piece of it.  And the thing about these kinds of syntactic improvements is they snowball, right?  Because that next Fib pair function is so short and so beautiful now,  I'm not going to bother to give it a name, right?  I'm going to drop it down to where I'm actually doing the job.  And that's the kind of thing that allows you to make categorical improvements in the readability of your code.  Because now you can say, now you have control of locality, where I had things that were apart,  I've now brought them together.  I'm not saying you should bring everything together.  I'm not suggesting one giant do all function.  But I am saying that having the ability to choose where to bring things together allows you to write expressive code.  Associative destructuring works the same way.  We bind names by keyword lookup.  I'm not going to look at this.  Let's get past this.  The only other example of a little bit of declarative programming that I want to show you,  this is another little DSL that's in closure,  is solving the actual problem of Lisp for beginners.  The actual problem with Lisp is not parentheses.  I already showed you that there are less parentheses in idiomatic Lisp  than there are in idiomatic Java or C-sharp.  The problem is things look inside out.  And I'll show you how that starts to happen.  So here I have a map.  Jonathan's password is secret.  And now I want to associate something new onto it.  So notice a social appears outside it and the nickname John appears on the other side.  So I'm building up on the left and right of the thing I started with.  I'm building out.  And then I decide to dissociate off the password because I don't want anybody to see that.  And now you start to see this thing that builds up inside out.  And most programmers, most humans, don't like to read things inside out.  How do they like to read things?  Left to right.  So Clojure has a macro called thread first  that reorders programs from inside out to left to right.  There's another macro that reorders things from inside out  based on the last argument instead of the first argument.  So there's actually two flavors of this.  But the point is, I guess there's a couple of points.  One is, it's one thing to say I don't like inside out reading,  but I want to have a language where I can decide what order I do things in.  So in Clojure, I could choose to go left to right.  I could choose to go inside out.  I could be perverse and implement my own macro that made you read things right to left  or top to bottom.  Or bottom to top. That seems a little crazy.  But the thing about this that's important other than just doing it  is that these capabilities are built in Clojure.  So this is extending the language itself by making these changes.  So these DSLs are what you'd call internal DSLs, right?  You can do this right in the language.  A second thing that's going to make me feel functional  is having a powerful set of persistent data structures.  We've already talked about this.  But I skipped one on purpose.  I said we're going to look at maps, vectors, sets and lists,  but we didn't actually. Do you remember which one we skipped?  I showed you API for three of the four.  A lot of stuff coming really fast.  So the one we skipped is lists.  I didn't show you any functions for manipulating lists.  And the reason that I didn't is that in Clojure,  you don't actually think about manipulating lists.  You think about sequences as an abstraction over any kind of collection.  And so the sequence library,  although the syntax of the things that are going to be returned,  they're going to print with parentheses around them,  so you're going to go, oh, that's lists.  They're actually not. Sequence is an abstraction over any collection.  So sequence is built at the bottom on first, rest and cons.  First returns the first item in a collection.  Notice that I'm not using it with a list, though, to my point.  First is working here with a vector.  Rest returns the rest.  And cons sticks something on the front.  Take and drop are recursive generalizations.  So take2 gives me the first two things.  Drop2 gives me everything else after I skip past the first two things.  I can apply predicates and higher order functions.  So I've got every, not every, not any and some for picking apart collections.  And my collections can be lazy or infinite.  So you already saw making the Fibonacci's.  Here you have the whole numbers.  Ink zero makes the whole numbers.  Cycle12 returns me an infinite collection of 12, 12, 12.  RepeatD returns an infinite collection of Ds.  I don't actually have infinite business data,  but it is nice to be able to treat things.  I mean, where do you think infinite?  Actually think streaming through data larger than memory.  I have a big data job and I want to stream through it.  This is really powerful.  And of course, map, filter and reduce.  So here we have another API, range,  which gives me a range up to a number.  So range10 is zero through nine.  And then I'm showing you map, filter and reduce here.  So filter subtracts out of a sequential thing,  stuff that or actually keeps in a sequential thing,  things that match a predicate.  So keep out of this sequence all the odd things.  Map, sometimes called collect or select,  depending on your language, says pluck.  Call this function on every item in the collection to give me the results  and then reduce walks a collection with an accumulator.  And so I'm just reducing plus.  Now, the big thing here is not that I work with lists this way.  The big thing here is that I work with everything this way.  So CECs work with collections.  They work with directories on the file system.  They work with the contents of files.  They work with XML data.  They work with JSON data.  They work with results that's coming out of a database.  Just to give a slightly larger example,  I downloaded the Rotten Tomatoes API yesterday in the hotel room  and I asked myself the question,  if you look at the movies that are currently topping the charts,  what actors are in more than one?  So here's how I'd go about doing that.  I'd find the JSON API, I'd download it, I'd parse the JSON,  I'd walk the movies, I'd accumulate up all the cast members.  I would extract their name because the data structure has in it more than names  and I just asked about names.  I didn't get the frequencies.  So if I've encountered an actor more than once,  I want to have a number that says,  oh, I encountered that guy twice or ten times or whatever.  And then I'll sort by the highest frequency.  So there's my pseudocode.  It's eight lines.  There's the actual code.  So go to the box office, you're slurp it down, read the JSON,  grab the movies out of it, map cat the abridged cast into a collection,  map the name out of that collection, grab the frequencies,  and sort by the negative of the second entry in the frequencies table,  which is going to sort them from highest to lowest.  Notice that I also naturally, as a closure programmer,  generalize the question.  I'm always going to generalize the question and return an infinite sequence,  a potentially infinite sequence, because the consumer can then just take the part that they want.  Right?  They can just go and grab the little bit that they want.  And if you're curious, as of yesterday,  nobody was in more than two movies that were in the top 50 from the API,  but there were a bunch of people that were in two.  Yes?  So this is a question that comes up commonly, particularly in a statically typed scenario.  So I broke the presentation of the API into two pieces,  specifically to try to cover this, and I'll be explicit about it here.  All of the collection types have APIs that are about them and return them.  So map APIs that return maps, vector APIs that return vectors,  set APIs that return sets, and then the sequence abstraction  takes you into a different world and says,  I want to process this sequentially.  So you can do either or.  You just have to know which one you're doing.  So how do you make a map if you have a vector, for example?  So there is an eager map called map V.  So map V is, and we don't care about the concrete type.  It's not how we roll, right?  We care about the semantics.  And so map is lazy and map V is eager.  But so if you wanted to do, you do want eagerness, though.  That's right.  And so map V will take you to that data structure.  So there are paths to what you want,  but they're not the most common and enigmatic thing.  But it's not forbidden.  You'll want it less once you do this for a while, though.  So I also want to feel logical.  And I'm not going to go through these examples in any detail,  but the point that I want to make is that  Closure has been a community that is incredibly friendly  to adding logic programming.  And so logic programming is not,  I'm going to go over and invest in a prologue system, right?  It's something that just happens naturally inside  of a closure programming.  There are at least three fairly well-established libraries  for doing logic programming in Closure,  Cascalog, CoreLogic, and DatomicDatalog.  Cascalog, this looks like a query.  This says, walk the people data set,  pulling out name and age where age is greater than 40,  and then walk the people data set,  pulling out name and age where age is greater than 50.  As an example of two different queries,  this is not super interesting.  I just copied it off the website.  But the actual thing about this is cool,  is that this is backed by Hadoop.  So this is an extremely powerful small set of lines of code.  This says run a parallel query on data sets  that could be spread across tens or hundreds of machines  and give me back the results.  CoreLogic is, this is the classic monkey needs to move a box  to the middle of the room problem implemented in CoreLogic.  I would encourage you to go download CoreLogic  and take a look at it.  The work on CoreLogic has been so appreciated  in the logic community that the closure community  and the scheme community are starting to have a little love in.  And the scheme conference next year is actually going to be  co-located with the closure conference.  And it's been super exciting to have really high powered  academic schemers and particularly reasoned schemers,  people who are doing logic programming and scheme,  sharing ideas and sharing concepts with the closure community.  And then here's Datalog in Datomic.  So this is the equivalent of SQL,  kind of in our world.  You can think of expensive chocolate and related product as views.  And when you see variables repeated in a query,  that causes a join.  The thing I like about Datomic Datalog,  and I have a personal interest because I helped build it,  is that it is just beautiful to read,  to do big jobs compared to SQL, right?  There's no joining through tables and getting caught  in the structure of the join tables  and all that sort of muck that you have to do.  And all of this stuff is not leaving the language, right?  This is not an alternate world that has to be magically brought  into the language.  This is just people building libraries in the language.  Finally, I want to be effective.  And I mean the word effective here in the literal sense.  I want a program that has effects.  This is where functional programming sometimes runs aground.  I mentioned before that one of the things people do  in functional programming is they sort of wave their hands  at this part of it and say,  well, you know, you're in Scala, you can say,  well, you use functional where it makes sense,  and when you need to have an effect, do you have an effect?  You do something different.  You can take the high road with types  and take a monadic approach to effect.  Or you can do what closure does,  which is to get away from these in place effects  by substituting value transitions.  So here's where we are today, right?  Every little bit of our program is a machine  and putting together programs is like assembling things,  all of which are moving.  So it's no wonder that it's hard to do, right?  Composing programs is really difficult  because everything is a bunch of moving parts  that you're trying to stick together.  And this made sense when memory was at 1970s prices.  In closure, new memories always use new places.  Memory is not that cheap anymore,  and we can let garbage collection forget  whichever old memories we don't care about.  So change is encapsulated by constructors,  references refer to a point in time value,  and references only see a succession of values.  So here we have three values, v1, v2, and v3.  We have constructor functions that know how to make values,  that maybe take an old value and give you a new one.  And then we have atomic succession from one value to another.  And then a view of this across time,  once upon a time the New York Yankees had this roster,  and then somebody got fired and they had that roster,  and then two people got hired and they had that roster.  Those values are protected by this reference,  and observers who are outside that can perceive the identity  and remember it and record it,  and they can use it in a concurrently safe way.  And this is the big one, observers do not have to coordinate.  So observers of references or values  don't have to coordinate with each other.  And people look at this for the first time and go,  aren't these just variables? No.  They're not just variables.  They represent understood semantics  of transitions from value to value.  If you've ever had to write a lock,  then you're doing something different  than what we're talking about here.  These things do not have to be protected.  Here's what it looks like from the API level.  I have a reference constructor.  In this case, I'm using one called Adam.  That makes a reference around the value zero.  And then I'm going to take a pure function  to do something to the contents of that and some args.  And then I have an atomic succession function.  So swap says, send a function inside this box,  apply the function inside the box,  give me back the result.  So what will be inside of the Adam after this?  It was zero, now it will be 10.  And here's mapping that back to the picture.  So I have a reference.  That reference points to an initial value.  I apply a function.  We have an atomic succession of the reference,  and then it's at a new value.  I can do the same thing with,  this model is completely general.  I can do the same thing with bigger structures.  So now the Adam has in it different data.  It's actually the equivalent of a Java bean or something.  I'm creating a person, and I'm swapping,  associating in a different name for that person.  I can also have varying semantics.  So now instead of having Adam,  I have a different kind of ref called a promise,  and a different kind of succession function called deliver.  So there's not a single semantic for concurrency.  There are five or six or seven,  depending on how you count them,  and they are all safe.  This model also scales all the way up to an entire database.  So here, I have, this is not closure anymore.  This is datomic, which is built in closure.  And I'm connecting to a database and transacting against it.  Transact is a succession function,  just like all of those other succession functions.  It's acid, and you start with a value of the database,  you get back a new value of the database.  In fact, the return value from a transaction is a future.  So you can decide whether you want to wait on the return value or not.  And inside that future, you have the old value of the database  and the new value of the database.  It is this atomic succession model writ large.  Now, the obvious question, and this came up on the panel,  really, you give me back the entire value of the database?  What happens if the database is big?  What happens if the database is ginormous?  And the answer is that these databases are values.  So I don't have to give you back,  I can evaluate the database lazy.  I don't have to give you back the entire database.  What can I give you?  I can give you the promise that I still know how to get all the rest of it.  And then I can fetch the other pieces as needed.  So you can grab these slides after.  There's a breakdown reference of the various other reference types,  but we're not going to look at them today.  The last thing I want is for my software to be pervasive.  I don't want to have a private island runtime  that I have to go and convince managers to install.  I want my systems to run on the platforms that people use.  And I think those platforms for language like Clojure  are the JVM, JavaScript, and.NET.  Clojure was originally written for the JVM,  that is the most mature implementation,  that's where the most work has gone forth.  Clojure CLR came next,  but Clojure Script has already passed it probably,  if not in maturity, at least in widespread use.  And the problem, unfortunately, is,  and you know this as well as I do, most of you in this room,  that languages on the CLR languish until and unless they're blessed by Visual Studio.  So I think that Clojure is backwatered on the CLR  because there's a strong commitment to using those tools,  and the tools are not available.  And I would say it would be great if some of you would take it out for a spin  and help make Clojure on the CLR more mature  and take it in the direction of use that it has seen on Java.  So in conclusion, everybody in the industry almost  builds things out of transient data structures that don't compose.  This includes Java, C-sharp, relational databases, and no SQL.  Clojure solves this with persistence and succession of values.  Persistence gives us systems that are composable  because the parts don't move around while you're not looking.  Value succession gives us a way to actually model state that makes sense in that world.  There are other choices. You can go down the Monad Road, for example, if you want.  These better semantics allow us to write more powerful and more flexible programs,  and those semantics are available on real-world platforms  where you want to build and run software.  I've included at the end of the slides links to a bunch of references.  One of these references, and you can easily Google for this.  If you Google for Stuart Halloway presentations, I actually have a wiki that's got slides  from all the presentations that I do, and I have left, I believe, four minutes for questions.  Yes?  You mentioned using an outside tool to use the application of the application.  Do you have to use a real-world?  Yes, so the question is, what is my outlining tool?  Yes, I use org mode and snippets.  I totally get that that's not where the world is going to go,  and I don't want to be an Emacs user.  I'm just forced there by its compelling features.  But I understand that, you know, telling someone to use Emacs,  if you're not already an Emacs user, did you just tell me to F myself?  Right? Because it's a hard learning curve.  There's IDE support for closure in all of the mainstream Java IDEs,  and it has characteristics that are more familiar and friendly to developers  that don't matter very much to me, and what it's missing is general text editing capability,  speed, and in particular, a structural editing mode called par edit.  So if you've never worked in a language that had a structural editing mode,  I would encourage you to do a Google search for par edit, P-A-R-E-D-I-T.  That's the big reason that most closure users stick with Emacs once they get hooked on it.  Other questions?  Those of you who are going to be watching this on camera,  I'm in a brightly sunlit room, shading my eyes so I can see people.  Yes?  So the question is, is it possible, given that the data structures evolve by creating new immutable data structures,  can you get back to the old ones?  The answer is you could if you wanted to, and it depends.  So closure does not, by default, keep the last ten versions of a ref and let you go back to them,  but the ref semantics, the model of refs, you could write a ref that had those semantics  if it made sense to do in your program.  Also, obviously, if you hold on to a value, if you go back and look at my picture of the observers,  some of them are looking at old value, as long as you're still looking at it, you're still looking at it.  Now, on the durable side of persistent data structures, when you go to Datomic,  that works a little bit differently because in Datomic, everything is time-stamped  and nothing is ever thrown away in that sense.  So in Datomic, you can actually put a filter on the database that shears off data from some time range.  So you can say dot as of a month ago, teleport your database back in time a month and query against it there.  That's the kind, and it's a great example of the kind of thing that's theoretically made possible by this.  Not all the reference types deliver that capability, but it's possible.  One last question.  Okay.  Right, so how does this data structure copying thing work efficiently?  So if you imagine a linked list, if I have a linked list with four guys on it and you stick a guy on the front,  I don't know you did that, and that's called shared structure.  This guy's talking to that guy and nobody cares.  The data structures in Datomic are all listed in sideways.  They're all trees, and they have the same property.  So you have an associative data structure over here that's represented as a tree.  You make a change to one thing. You get a new tree that's pointing willy-nilly all into the old tree to reach all the old stuff except the one.  So the cost you pay is what's called a path copy, and in closure, the paths go five deep.  So it's actually, and they are, it's log 32 in instead of constant time,  but given that it only goes five deep, it falls out in the wash.  In Datomic, on the durable side, those trees are three deep.  So they're incredibly wide branching trees, so you don't pay a big penalty for that.  That is slower.  Those data structure updates are slower than a purely mutable data structure.  So on the flip side, you get the speed back on reads, right, because you don't have to coordinate, you don't have to lock.  And so if you live in a world where there will be more multi-core processors in the future, closure will become better and betterer.  And if you live in a world where multi-core processors are going to be defunct,  then the stuff I'm talking about is ridiculous and you should ignore me.  So my name is Stuart Halloway.  Make sure you remember to fill out eval cards.  I have discount coupons for this O'Reilly closure training video and closure stickers.  If anybody didn't get those, come and see me after the talk.  Thank you very much.
Stuart_Halloway_-_Introduction_to_Clojure-68375202_0_191.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_192_239.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_240_671.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_672_6251.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_6252_9731.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_9732_17507.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_17508_18587.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_18588_26711.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_26712_31211.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_31212_40931.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_40932_41339.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_41340_41771.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_41772_41987.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_41988_42035.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_42036_43079.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_43080_43127.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_43128_43271.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_43272_43343.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_43344_43415.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_43416_44507.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_44508_44579.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_44580_44771.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_44772_44939.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_44940_44987.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_44988_49019.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_49020_49067.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_49068_49715.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_49716_58055.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_58056_65807.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_65808_66131.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_66132_68147.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_68148_68195.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_68196_69047.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_69048_69095.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_69096_70031.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_70032_70763.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_70764_70823.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_70824_72683.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_72684_84047.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_84048_84311.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_84312_84743.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_84744_85943.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_85944_86999.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_87000_88403.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_88404_89843.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_89844_91571.png Stuart_Halloway_-_Introduction_to_Clojure-68375202_91572_91603.png
Good morning everybody. I'll try that one more time. Good morning everybody. They're all out there on the audio. They just haven't had their coffee yet. My name is Stuart Halloway and I'm going to be talking to you today about closure, which is a programming language for problem solvers. And I want to start from first principles and talk about what I want software to be like. And this is something that we're, you know, working towards gradually. I want software systems to be knowledgeable, right? I want them to have valid and correct information about the world and tons of it. I want them to be powerful. And I want to feel powerful as a programmer using it. You get that high when you do something on the computer faster or easier than you thought you could before. It's just an amazing feeling. I want software to be flexible. How many of you have had the experience as a developer of having a business stakeholder come to you and ask for some feature which sounds really incremental? And you had to say to them, eh, I know that sounds really incremental, but actually given the way the system was architected, it's seeing the impossible. Or it's going to cost doubles. Anybody had that experience? That's a terrible feeling. I don't want that feeling. I want the opposite of that feeling where I make the software 10% more capable with 10% additional effort. And that's very tricky to do. I want my software to be smart. I want programs to anticipate my needs, to be out ahead of me, to have capable analysis. I want them to be effective. I want the decisions made by software to actually enact change in the world. Change in my world, change in the business world. I don't just want idle information, I want action. And finally, and this is really something that has become dominant in the past couple of years, I want software to be pervasive. I and all the users that I write for have an expectation that their software is going to be available how many hours a day. 24, right? Everybody expects 24-7 software. And where is the software going to be available? It used to be software ran in big installations. And then we had our own desktop machines. And then we had laptop machines. And now I have an expectation that I can carry a device in my pocket or on my wrist or in a bracelet that's going to be more powerful than the software that put man on the moon. And if that's not true, I'm going to be pretty disappointed. Now, we try to build, this is, and you know, I would say as a guy who's been in the industry for a while, this feels a little bit like scope creep in terms of what our users want. There was a time when we could have made them happy just having anything at all. But they get more and more demanding. Now, we have to build software in a world where we operate under certain physical laws. One of the physical laws that we operate under is that memory is very expensive. In particular, you probably wouldn't want a provision system with more than, I don't know, 100 kilobytes or so. Because it would be extremely costly to do. Secondly, storage is really expensive. And it's really slow and it's not random access. You have to get bits and bytes off of tape and put them back. So that's a big constraint that we operate under. And machines are really special and precious. Machines are not these anonymous things that you can trivially stand up and down. They're these expensive constructs. They're so special and unique that they're dedicated to tasks. We live in a world of dedicated machines. There's a special database machine somewhere. I know it by name. I call it Gandolf. You know, I get to know it. I swap its hardware out over time. And so, what's wrong with this slide? When was this true? 30 years ago. 30 years ago, right? I think, and, but the problem is that we are working with languages and technologies that are incremental improvements on designs from then. And basically what it comes down to is that the dominant paradigm for writing software in the industry right now is really the greatest hits of the 1970s. And this is a big problem and it's what we're trying to solve with closure. One of the problems that the 1970s give us is an approach to data structures that I'm going to call transients. So if you look at the way we store data, we have to store data transiently because memory is expensive and because storage is expensive, right? Remember, we can only have a couple hundred K of memory and, you know, storage is really pricey. And the effect of having storage be transient is that we have mutable data that we bang away at in place. And so, you see the effect of this on our building systems in a bunch of ways. If I have a data structure which is transient, I can't easily or comfortably share it with you, right? Because somebody might be changing it. It might erode out from under them. Likewise, it's difficult for me to distribute something that's transient because if it's transient, something's happening to it. And we all got to go back to the happening place, right, to find out what may have happened to it. Obviously, concurrent access of transient data structures is a catastrophe. Transient data structures have to be consumed eagerly because if you consume them lazily, by the time you get to them, they may have changed. And they're difficult to cache. Has anybody ever worked on problems of cache invalidation? You can't invalidate something that's persistent, but transient things invalidate all the time because they're changing out from under you. And everything is built this way. The collection APIs in Java are based on transient data structures. The collection APIs in.NET are based on transient data structures. Relational databases are based on transient data structures. And no SQL databases are based on transient data structures. So while I'm excited about no SQL as a movement, it hasn't really addressed what I consider to be the biggest problem yet. And all of this transience comes from a set of design principles that made sense when memory was a million times more expensive than it is now. Imagine how much your world would change if something else got a million times easier to do. Imagine if travel got a million times faster. Right, I live in North Carolina in the United States. It would categorically change the way we planned and organized events. If I could move back to the United States a million times quicker than I will, I'd in fact do tomorrow. We've had that million fold change and very few people have gone back and looked at how we write code and said, hmm, maybe we should do something different. Now, this transient data structure problem is the most important single example of a more general problem, which is complexity. And you know complexity by its symptoms. You have a nonlinear difficulty maintaining systems as they grow. Right, and this could be by any definition of growth. This could be by volume of data. This could be by volume of activity. This could be by size of code base. This could be by features. We've all had that experience that the system gets twice as big and how much more costly does it get to maintain? Even worse, we've had that experience of having the inability to make incremental change to a system. How many of you have some piece of your system that you're terrified to touch? Because you have no idea what's going to happen after you touch it. I mean, our programs are starting to be like the weather. Oh, I see the software is raining today. I have no idea how or why that happened, but I guess we'll just have to live with it. And then of course, in the limiting case, you have large efforts that fail entirely. And so you've probably already heard people throw about statistics about 50% of large IT projects fail, blah, blah, blah. Maybe you've had some experience with that. So we want to attack these problems, and closure is an effort to attack these problems. And it does it in several ways. The thing is that closure uses persistent data structures. I can tell you right now that if you learn nothing else from this talk, if you learn nothing else today, the most important thing you could take back to your job if you don't already use persistent data structures is to start using them. This is a language agnostic observation. You don't have to do the F-sharp pass persistent data structures. So there are places that you can go and get this regardless of what platform you're on. This is where the world is going for all the reasons that I said in the table on the previous slide. Closure is powerful and flexible. I feel like I'm able to get more done, and I feel like once I've gotten something done, I can change it and adapt it to my needs. One of the things that falls out of persistent data structures is functional programming. If I'm going to use persistent data structures, everything's immutable. And so I live in that world of FP. And where that tends to run aground is that we do want our programs to have effect in the world. And a purely functional system would just sit there and get warmer. It wouldn't have any input or output. And so there's a bunch of ways people deal with that problem in the functional languages. One way is to model time actually as functionally as well and use monads. And so there are functional languages that go down that road. Another way is to sort of punt a little bit and say, well, you know, use functional when it's suitable and use something else when it isn't. A third way, which is what closure does, is to say that identity in a system, something that's changeable over time, changes via succession of values. So you have an immutable value, and then you have another immutable value, and then you have another immutable value later. Closure is a Lisp. It's actually the least important of the points on here. Obviously, that hits you in the face when you first look at it, but I made the observation on the panel discussion earlier in the week that someone said, do we have too many functional languages? And I said, I don't think we have anywhere near enough. Just to pick one example familiar to heart, you could build a language based on all these principles except for Lisp, and you would have a language that doesn't exist in the current ecosystem. There's not a choice out there that has closure's characteristics minus being a Lisp. Finally, closure is designed by and for professionals for building professional software. It is not designed for hobbyists or entertainment. It is not designed for newbies. It is not designed for academic use. People do all those things, obviously, but when the rubber meets the road and decisions have to get made in the language, the language is squarely aiming at expert professionals getting shit done. So let's look at the language. Closure is built on top of a data format called the Extensible Data Notation, or EDEN. And when you see EDEN in a moment, you're going to think JSON, and it is similar superficially to JSON, and we would have used JSON except we needed the following three characteristics which JSON does not have. We needed a richer set of built-in data types. JSON has string, which ends up having to get multipurpose for a lot of different things. We wanted a generic mechanism for extensibility, so the ability to add new kinds of data structures without requiring existing serializers or consumers to have to know about them in order to do their work, and language neutral. And we didn't want to get into the object game at all. So I sort of think of my experience with data serialization as a sort of Goldilocks and the Three Bears kind of story where, you know, XML is too hot and JSON is too cold and EDEN is kind of just right. And the sense in which it is just right is that it is richer in its data capabilities than something like JSON, but it does not run off the rails in trying to pursue identities or objects or things like that. And for those of you who have done this stuff for a long time, if you remember things like SOAP Section 5, that's the kind of catastrophe that I'm talking about. So let's take a look at EDEN notation. We have strings, they're double-quoted. In Java or C-sharp, there will be strings. We have characters led by a backslash. There will be characters. We have integers, doubles, booleans, true and false. We have nil. The word nil as opposed to null comes from the Lisp world. We have ratio types, and then we have two types that can represent names. We have symbols, which just look like words, and there's a rich set of legal characters, so, you know, plus sign also could be a symbol. And then we have keywords, which just look like words with a colon stuck in front of them. Most of this lines up fairly well with syntax in C-sharp or Java or JavaScript as far as it goes. Then there are four collection literal types. A list is demarcated by parenthesis, and it suggests something that's going to be seen sequentially. So there's a presumption when something is serialized as a list that when you read it back, you're going to get something that is O of n to traverse it. Vectors, on the other hand, are associative collections by index. So the presumption is if you have something that's in the square brackets, that's something that you could efficiently look up the guy at the end in O of one time. Maps are key value pairs. So here we have the keyword A pointing to the value 100, and the keyword B pointing to the value 90. And sets are collections where you have membership. So here we have a set containing the keyword A and the keyword B. I will mention, and you see it here on the slide, commas are white space. So if you want to, you can go back and add commas into this code. And maybe when you get started, you know, you're initially programming and closure or writing Eden data, you will. And then you realize all the time you can save by never having to type a comma again, and you'll stop doing it, or at least most people do. Now, the funny thing is that I have just taught you the closure programming language. I've taught you the entire syntax because as a list, closure programs are written in data. They're not written in text, and they're consumed as data, not as text. So we take data in Eden, it's actually a superset of Eden, deserialize that into data structures, and that's actually what your program is made of. So here's Hello World. If I interpret Hello World as a piece of data, I see a list containing a symbol, printlin, and a string Hello World. If I interpret it semantically, I say, oh, the first symbol in a list is the name of a function to be invoked. So that is printlin Hello World. Now let's define a function. So here, this is the canonical syntax for defining a function in closure. Def in, define a function, greet the function name, a documentation string, an argument list, in this case a single argument, your name, and then stir, in this case, is taking two arguments and sticking them together and making a string Hello your name. But notice that this has not introduced any new syntax. I could also analyze this purely as data. So this is a list containing a symbol, a symbol, a string, a vector with a symbol in it, and a list with a symbol, a string, and a symbol. It's all made out of data. So it's 9, 14, and 52 seconds. That's how far we are into the talk. You now know closure. So we're just going to take a deep breath for a minute and enjoy the fact that it was so easy to learn the language. So now let's get back to those benefits that I wanted to acquire as a software developer when I'm building systems. I want to be power or knowledgeable. And to be knowledgeable, I actually have to have knowledge. I would submit that C sharp and Java and relational and no SQL systems actually don't have knowledge in them. They have machines in them. And you can go and look at the machine, and then you can come back later and look at the machine and what will have happened. It will have moved. It will have moved into some different position. And persistent data structures are a different approach to that. They are immutable, so they can't change. The only change in the world is by function application. Another way to say this in an object-oriented setting is the only way new stuff happens is via constructors. So as a sort of capsule way to do it. Imagine a world where no objects had setters ever. Everything only had getters, and you only made new things via constructors. That's what we're talking about here. Now, of course, in order to do that, we can't be naive about it. And so we have to have good performance, which means that these persistent data structures actually share structure internally. So I'm not suggesting that if you have a map with 100 items in it, and then you create another map that's exactly like that map, but with one more, we now have 100 more things in memory. Those things can share data together and maintain full fidelity access to old versions. So let's go back and look at the characteristics we get out of making these choices. Sharing transient data structures is difficult. They might change out from under you. Sharing persistent data structures is trivial. If I alias something and 100 people are looking at it, what could possibly go wrong? They can't change. Distribution is not trivial, but it goes from being difficult to easy, particularly when you combine it with something like content addressing. I can take big data structures, and we can share them and refer to them by their hash if we want. And once you know you've got it, you can cache it forever. Concurrent access is also trivial. Everything in Clojure is safe for concurrency and safe for being looked at by multiple observers. The access pattern is your choice. You can consume things in an eager fashion or in a lazy fashion. And obviously, caching is easy because things can just stick around forever. What things look like this today? A lot of functional languages support this, so Clojure, F-Sharp, and on the database side, Datomic, has the entire database as a persistent data structure. Now, having made data this way, we want to have power over our data. In the OO world, we get power over our data by having a proliferation of concrete classes. So I'm going to have the person being and the account being and the order being and the order line item being, that sort of thing. In a functional language, we follow more with Alan Perlis, who said, it's better to have 100 functions that work with one data structure than to have 10 functions that operate on 10 data structures. Or in OO, what do we end up having? Is three or four or five functions on thousands of different data structures. And so every time you sit down to write a program, there's a brand new API that somebody made up. In Clojure, every time you sit down to write a program, there's an existing API, and the API is Vectors, Maps, Lists, and Sets. So let's look at those. We'll start with Vectors. Here's a short program that defines the name V to be a vector containing 42, the keyword rabbit, and a sub-vector 1, 2, 3. I should mention, by the way, here, that everything nests. So all the different data structures you've seen nest inside each other arbitrarily. That's generally not true in programming languages that have a lot more syntax. Go to the top of your C-sharp or Java file and write a for loop around your import statements. And you'll get an idea of the kind of thing that I'm talking about. Because don't nest, there's rules about what you can put where, and you get in trouble, and the compiler will not let you move forward if you don't do that. So now having created an associative data structure, associative data structures act as functions. So instead of saying get from V, the item at the index 1, I just say V1. The first thing in V, and it's the one thing in V, obviously we're zero based, so it's not going to be 42, it'll be rabbit. I can peek at a vector, in which case I will see the thing at the end, 1, 2, 3. I can pop a vector, in which case I will see, and that should be popped off, so that the 1, 2, 3 should be gone. And I can take a sub-vector of a vector, so here give me the vector starting at 1, so 42 falls off the front. All of those things are constant time operations. Now let's take a look at a map. So here's a map with three key value pairs, A1, B2, C3. Again, maps are functions, and so instead of saying get from the map B, you just say map B. Keywords are also functions that look themselves up in collections. So everywhere in an object-oriented language where you'd be saying dot get first name, dot get last name, in an idiomatic closure program you would be saying colon first name in function position, the first slot in that list, and then say that guy's going to be 2. I can get the keys out of a map, so there's ABC, you can also get vowels, that would return 1, 2, 3. I can make new maps, so here's an example of one of those constructors. ASSOCI takes a map and some key value pairs and returns a new map. So here I'm taking the map M, ASSOCIing in D4 and C42. Now C used to be 3, obviously that gets overwritten. Although we should be careful about the use of the word overwritten. What happened to M when I did that? Absolutely nothing. I returned a new object that had those characteristics. M is still as it was, and I can dissociate keys from a map. So dissociate D from M. And then you can get to some more advanced ones. So merge with is a function that takes a function. So this is a higher order function when people talk about functional programming doing higher order functions. This says combine M with A2 and B3, but instead of overwriting values where they collide, combine them using the combining function passed into merge with. So in this case merge with is being the pasta plus sign. So it says oh I see, A used to be 1, but I passed in A2. I combine those, 1 plus 2 is 3. And you can go further with dealing with say nested structures. So let's imagine I have a nested structure. John Doe has a name, but John Doe has an address, and the address is actually nested further. The address has a zip code in it. There's a function called get in that will take a data structure and a list of keys and walk the path of keys down to what you're looking for. So that says walk the path of keys from address to zip, so that's going to return 27705. You can also use this to update data structures. So assoc in is the recursive generalization of assoc. It says take this data structure, drill down into it somewhere, and then make a change at that place. So now I've changed the zip code to 27514. And perhaps even the more interesting one is update in. Update in takes a data structure and a path and a function. And it applies that function to whatever finds in the path. So here I can say go to JDoes, address zip, and increment it. So it would go from 27705 to 27706. In a certain sense this is trivial. In fact, it is trivial. On the flip side, if you built an OO program, there would be a special class that represented the person and a special class that would represent the address. And then it would just be busy work to navigate through it because there wouldn't be a generic access path like this. So let's look at sets. Closure has some extensions for set in the set namespace. So that loads an additional namespace. You don't need this for the literals, but you need it for some of the functions. I'm going to define the colors to be red, green, and blue, and the moods to be happy and blue. I can disjoint from a set. So if I disjoint red from red, green, and blue, I'll get back green and blue. I can take the difference between two sets. So the difference of colors and moods subtracts out the stuff in moods. So I subtract out blue and then up with green and red. I can take the intersection of colors and moods. And I can take the union of colors and moods. Not shown here, but I leave it for you to go play with. How many people feel like they have a grasp of the relational algebra that underpins SQL? Anyone been there, done that? So closure.set the namespace implements the entire relational algebra on Closure's data structures. So you can just go in there and party on and project and join and whatever you want to do. So that's generic data structures. I want to feel powerful. And this is pretty darn easy to summarize. The way I want to feel powerful is I want to have bare metal access to my platform. A lot of people mistakenly divide the world into two kinds of languages. Languages that are at the metal platform languages that are more than highly expressive, friendly, or to the program languages. That's a false dichotomy. I want both, please. And so closure provides direct access to the platform. You can construct things on the platform. Most platforms, it's going to be new class name, some argument. In closure, it's always the verb first. One of the things about closure is all action in the world always comes in the first position of a list. So when you get ready to start metaprogramming, it's a beautiful thing. Everything works the same way. So making a new thing, you see the class name and function position. That says make a new red widget. Static members, math.py on the platform would become math slash pi. Instance methods, rand.nextint would become.nextint on rand. Notice again, perfect consistency across the language. Verb always comes first. Grabbing an instance field would be pixels.data on the platform and would be.data pixels on closure. You're not going to see this one very often because even though I wish they would, idiomatically Java and C-sharp programmers don't encourage you to access the fields of their classes. They tend to make them private. The place where you will see that syntax quite a bit is if you're writing a closure program that's targeting JavaScript. Because in JavaScript, people do that all the time. Yes. So there are three platforms, I think, that are super important. And they are Java, the CLR, and JavaScript. There could be others, but those are the three big important ones. When I say bare metal, I mean access to the platform. I mean specifically, if I'm writing a Java project, I don't want to have to drop to Java to do some high performance thing. Now, I am not talking about dropping underneath garbage collection. So I'm not talking about dropping to J&I or whatever the platform access called in C-sharp, I can't remember. Yeah, PNVoc. I'm not talking about dropping to that level. And I'm not talking about competing with the C guys for their bare metal. So I'm not going there. I am talking about platforms that are garbage collected in virtual machine platforms. So chaining access on the platform, it's x.y.z. Enclosure, it's.. at the beginning. And then that says, that's the recursive generalization of do something, right? So person.getaddress.getzipcode. You don't have to write the. again and again. I will point out that a lot of people, when they first look at Lisp, they make this snide joke on Twitter about punctuation. But if you go back and look through these slides, closure has fewer parentheses than the same code in C-sharp or Java would. And furthermore, it has no commas. And let me tell you, I have been writing code in an Algal syntax language for the last week, and it is murder to have to type commas. I feel for people who have to type commas when they're writing code. You can implement interfaces in closure with a form called reify. So this says reify runnable and implement the run method to print hello. And I have one example just to demonstrate my point about platform power. It's actually a JavaScript example. And you can go and read this article. Some of you may have seen David Nolan. He's here speaking at the conference. He's the lead developer on ClojureScript. And he has been doing benchmarking to verify that ClojureScript can do anything that JavaScript can do, so that there's no need to drop down to JavaScript. And so he put together this demo, which I will attempt to open. So this is a port of Notch's JavaScript performance demo. So this is a purely calculative exercise. This is a processor burning thing. It's doing a Minecraft simulation internally. And David has been verifying that ClojureScript is neck and neck with JavaScript at this. And this is even doing things like icky, mutable, imperative, at the bottom of the rendering level, loop stuff. So I would encourage you to check this out if you're skeptical about claims of platform performance. You could do similar experiments with Clojure running on the CLR or Java. Oh, and you know what? I need to turn that off because it literally will burn one core. So my machine would start to sound like a fan. Its job is to use a core as fast as it can. So I would summarize the Platform Interop story in Clojure as simple, wrapper-free. And this is a big deal. The minute you have wrappers talking to Java or CLR or whatever, then you're no longer in the world of getting the direct stuff. You're now in the world of having a value-added service, which becomes a value-subtracted service when you want to do something different. It's performant and it's conformant. It gives you access to the features of the platform. So I'm feeling pretty knowledgeable and powerful. I want to talk now about feeling flexible. And so a big piece of feeling flexible in a language like Clojure is that the work style is interactive. There's this website. You can go check it out called Tri-Clojure. Tri-Clojure runs a Clojure REPL in the browser. And you can go and just start programming. There's also a website called Himera. Himera is a Clojure script REPL running in the browser. Now, because we have only an hour today, I'm not going to do an extended demo of what it's like developing at the REPL, but I want to talk about it. I had dinner last night with Neil Ford, who will be speaking later today. And I was asking him, he's at ThoughtWorks. I said, how do people develop code in the agile world these days? What's the pattern? How do I move quickly? And he said, I think that it's primarily, it's not necessarily test-driven, but it's keeping the test green. It's having a fast feedback loop and you code and you see the test pass and you code and you see the test pass. That's the people who are working the fastest how they work. I don't work like that. The way I work, I would describe as literate planning. I'm people familiar with Nooth's literate programming, the idea that your program would be a document. There are a bunch of people in the Clojure community who are into that. I'm not going to say that I'm that far, but I do most of my work as a programmer, either in a drawing tool, an OmniGraphil, or in an outlining tool. And inside the outlining tool, I have snippets of code that I send to the REPL for evaluation and gradually build up larger and larger programs. I tend not to write unit tests almost ever, because by the time I've done exploring at the REPL, I've produced a pure function, which I've seen to work on a variety of inputs. Instead, I write generative and simulation tests, which are at a higher level. So generative tests will manufacture inputs of different kinds for functions and then verify assertions about those functions. And then simulation tests, really, I think that B's needs in testing, is having a database of activity simulating real-world activity against your system and running it that way. If you're interested in that, I am giving a talk on simulation testing. I think not in the next slot, but the slot after that this morning. But the point I want to make here is that this interactive style of development is incredibly rapid, and it's quite a bit different from IDE-driven development or test-driven development. In addition to that interactivity and being able to build a system quickly, I want to work in a dynamic language. So Clojure is a dynamically typed language. You notice that we had a bunch of data structures up there, but there weren't any classes or types or any of that stuff. And I want to show you what this feels like by refactoring a piece of common Java code into Clojure. So this method is from a library called the Apache Commons. If you have been in the Java community for more than five minutes, Maven, while it was downloading the Internet onto your machine, certainly downloaded the Apache Commons, because it's used, it's a utility library that does common things that Java doesn't ship with. And so this particular, you can try to read it, this particular function is a validator function that verifies that a string is blank. And the way that a string is blank is, well, if it's null, it's blank, and if it's linked to zero, it's blank. And then the complicated part, if it's got stuff in it, but all of that stuff is white space, then it's blank. So I'm going to take this idiomatic Java code, and I'm going to refactor it to idiomatic Clojure code. So I'm going to remove the type declarations, and this is what people think about when they're coming from a statically typed language to a dynamically typed language. They picture the code that they had with all the types removed, right? And I agree with you that that would be a catastrophe, right? If I stopped here, we would be in a bad place, because basically what I just said is you live in a tar pit with good maps, and now you live in a tar pit with no maps. So we've gotten rid of the maps, now let's get rid of the tar pit part. So I'm going to get rid of the class, because we're just going to make a function, and I'm going to introduce a higher order function. So we don't have a higher order function syntax in Java, I'm just going to make this up. So I'm going to say every ch and stir character is white space, and you can sort of imagine what that does, right? That's an expression that returns true if it's true for every guy in the collection. Now, one of the great things about this higher order functional style of programming is once you've written it that way, all those corner cases around it actually disappear. So now I have is blank of a stir is if every character in stir is white space. And then I'll actually drop down to closure syntax. Now, if you compare this with the strongly statically typed slide at the very beginning, this is the kind of advantage that I'm talking about of writing code in a dynamic language. And you end up with a very, very small number of very short, very general purpose things. And these pieces all have to fit together, because if everything wasn't shorter, that would suck, because I would still want to have the roadmaps that static typing gives me. I wouldn't want to go here. And if I had to make as many things as I would have to make to build an OO system, I'd still want the types. But I make 10 times fewer things on small closure programs, and 100 times fewer things on large closure programs than I would make if I had to write them in OO. I absolutely would. So the question was, would you say that it makes a difference which static language you're using? Yes, I think that the reasons that dynamic is better become much more subtle and nuanced when you compare it to something like ML. And I would be perfectly happy. And this is a point I should have made on the static versus dynamic panel earlier in the week. I think we should all get into functional, and then we can argue about the static and dynamic thing, you know, once we're at the end of the road. Because I consider that static dynamic debate is really kind of tiny compared to we need to be more functional and less imperative. And so I think that closure and F-sharp and Scala and ML and OCaml and all these guys, we should all, we're all on the same side here, right? We are all fighting for truth and justice together. So, intelligent. What makes systems intelligent? One thing that makes systems intelligent is being able to specify, do what I want or do what I mean, not how do you do this. And one aspect of that is being declarative. I'm going to give one example of a declarative DSL enclosure that's called destructuring. And the idea behind destructuring is I want to have a way to bind multiple names. I already showed you how to bind one name, right? DefX or letX, right? But I want to be able to bind multiple names, and I want to work against the abstract structure of what I'm working on. So I don't want this to commit me to concrete types. It's, again, another dynamic argument. And I want this to be available everywhere. So anywhere in the language where I would bind names, I want to be able to do this. And there's going to be two forms of it. I'm going to have a vector form, which is going to do sequential destructuring. And I'm going to have a map form that does associative destructure. This is probably the hardest really early roadblock for people in closure unless it's motivated. So I'm going to show an example whose intention is to motivate it. Let's imagine that I was writing a function to make the Fibonacci numbers. In order to make the Fibonacci numbers, I need to maintain two items of state, right? Because every Fibonacci number is built from what? The previous two. So I have to do something pair-wise and then maybe trim that down to actually see the Fibonacci's. So I've written this function next Fib pair, which takes a pair of numbers, and then it makes the second number, the first number of the new pair, and it makes the sum of the two numbers the second number of the new pair. And the problem with this, and then I iterate over it, by the way, iterate is a higher order function that produces an infinite collection. So iterate says take that seed and then do the recursive generalization of whatever the function is. So take 0, 1, call Nick's Fib pair on that, and then on the result, and then on the result, and then on the result, and then on the result. So that produces a sequence from which I can then extract the Fibonacci numbers. But the thing I don't like about this is that there's a lot of noise for picking apart the pair. And so what closure lets me do is you can add this extra pair of square brackets anywhere where you're binding names, like in a function definition, and that extra pair of square brackets says, I don't want to bind a name to the thing that I was given. I want to destructure it. I want to break into it, and I want to bind A to the first piece of it, and B to the second piece of it. And the thing about these kinds of syntactic improvements is they snowball, right? Because that next Fib pair function is so short and so beautiful now, I'm not going to bother to give it a name, right? I'm going to drop it down to where I'm actually doing the job. And that's the kind of thing that allows you to make categorical improvements in the readability of your code. Because now you can say, now you have control of locality, where I had things that were apart, I've now brought them together. I'm not saying you should bring everything together. I'm not suggesting one giant do all function. But I am saying that having the ability to choose where to bring things together allows you to write expressive code. Associative destructuring works the same way. We bind names by keyword lookup. I'm not going to look at this. Let's get past this. The only other example of a little bit of declarative programming that I want to show you, this is another little DSL that's in closure, is solving the actual problem of Lisp for beginners. The actual problem with Lisp is not parentheses. I already showed you that there are less parentheses in idiomatic Lisp than there are in idiomatic Java or C-sharp. The problem is things look inside out. And I'll show you how that starts to happen. So here I have a map. Jonathan's password is secret. And now I want to associate something new onto it. So notice a social appears outside it and the nickname John appears on the other side. So I'm building up on the left and right of the thing I started with. I'm building out. And then I decide to dissociate off the password because I don't want anybody to see that. And now you start to see this thing that builds up inside out. And most programmers, most humans, don't like to read things inside out. How do they like to read things? Left to right. So Clojure has a macro called thread first that reorders programs from inside out to left to right. There's another macro that reorders things from inside out based on the last argument instead of the first argument. So there's actually two flavors of this. But the point is, I guess there's a couple of points. One is, it's one thing to say I don't like inside out reading, but I want to have a language where I can decide what order I do things in. So in Clojure, I could choose to go left to right. I could choose to go inside out. I could be perverse and implement my own macro that made you read things right to left or top to bottom. Or bottom to top. That seems a little crazy. But the thing about this that's important other than just doing it is that these capabilities are built in Clojure. So this is extending the language itself by making these changes. So these DSLs are what you'd call internal DSLs, right? You can do this right in the language. A second thing that's going to make me feel functional is having a powerful set of persistent data structures. We've already talked about this. But I skipped one on purpose. I said we're going to look at maps, vectors, sets and lists, but we didn't actually. Do you remember which one we skipped? I showed you API for three of the four. A lot of stuff coming really fast. So the one we skipped is lists. I didn't show you any functions for manipulating lists. And the reason that I didn't is that in Clojure, you don't actually think about manipulating lists. You think about sequences as an abstraction over any kind of collection. And so the sequence library, although the syntax of the things that are going to be returned, they're going to print with parentheses around them, so you're going to go, oh, that's lists. They're actually not. Sequence is an abstraction over any collection. So sequence is built at the bottom on first, rest and cons. First returns the first item in a collection. Notice that I'm not using it with a list, though, to my point. First is working here with a vector. Rest returns the rest. And cons sticks something on the front. Take and drop are recursive generalizations. So take2 gives me the first two things. Drop2 gives me everything else after I skip past the first two things. I can apply predicates and higher order functions. So I've got every, not every, not any and some for picking apart collections. And my collections can be lazy or infinite. So you already saw making the Fibonacci's. Here you have the whole numbers. Ink zero makes the whole numbers. Cycle12 returns me an infinite collection of 12, 12, 12. RepeatD returns an infinite collection of Ds. I don't actually have infinite business data, but it is nice to be able to treat things. I mean, where do you think infinite? Actually think streaming through data larger than memory. I have a big data job and I want to stream through it. This is really powerful. And of course, map, filter and reduce. So here we have another API, range, which gives me a range up to a number. So range10 is zero through nine. And then I'm showing you map, filter and reduce here. So filter subtracts out of a sequential thing, stuff that or actually keeps in a sequential thing, things that match a predicate. So keep out of this sequence all the odd things. Map, sometimes called collect or select, depending on your language, says pluck. Call this function on every item in the collection to give me the results and then reduce walks a collection with an accumulator. And so I'm just reducing plus. Now, the big thing here is not that I work with lists this way. The big thing here is that I work with everything this way. So CECs work with collections. They work with directories on the file system. They work with the contents of files. They work with XML data. They work with JSON data. They work with results that's coming out of a database. Just to give a slightly larger example, I downloaded the Rotten Tomatoes API yesterday in the hotel room and I asked myself the question, if you look at the movies that are currently topping the charts, what actors are in more than one? So here's how I'd go about doing that. I'd find the JSON API, I'd download it, I'd parse the JSON, I'd walk the movies, I'd accumulate up all the cast members. I would extract their name because the data structure has in it more than names and I just asked about names. I didn't get the frequencies. So if I've encountered an actor more than once, I want to have a number that says, oh, I encountered that guy twice or ten times or whatever. And then I'll sort by the highest frequency. So there's my pseudocode. It's eight lines. There's the actual code. So go to the box office, you're slurp it down, read the JSON, grab the movies out of it, map cat the abridged cast into a collection, map the name out of that collection, grab the frequencies, and sort by the negative of the second entry in the frequencies table, which is going to sort them from highest to lowest. Notice that I also naturally, as a closure programmer, generalize the question. I'm always going to generalize the question and return an infinite sequence, a potentially infinite sequence, because the consumer can then just take the part that they want. Right? They can just go and grab the little bit that they want. And if you're curious, as of yesterday, nobody was in more than two movies that were in the top 50 from the API, but there were a bunch of people that were in two. Yes? So this is a question that comes up commonly, particularly in a statically typed scenario. So I broke the presentation of the API into two pieces, specifically to try to cover this, and I'll be explicit about it here. All of the collection types have APIs that are about them and return them. So map APIs that return maps, vector APIs that return vectors, set APIs that return sets, and then the sequence abstraction takes you into a different world and says, I want to process this sequentially. So you can do either or. You just have to know which one you're doing. So how do you make a map if you have a vector, for example? So there is an eager map called map V. So map V is, and we don't care about the concrete type. It's not how we roll, right? We care about the semantics. And so map is lazy and map V is eager. But so if you wanted to do, you do want eagerness, though. That's right. And so map V will take you to that data structure. So there are paths to what you want, but they're not the most common and enigmatic thing. But it's not forbidden. You'll want it less once you do this for a while, though. So I also want to feel logical. And I'm not going to go through these examples in any detail, but the point that I want to make is that Closure has been a community that is incredibly friendly to adding logic programming. And so logic programming is not, I'm going to go over and invest in a prologue system, right? It's something that just happens naturally inside of a closure programming. There are at least three fairly well-established libraries for doing logic programming in Closure, Cascalog, CoreLogic, and DatomicDatalog. Cascalog, this looks like a query. This says, walk the people data set, pulling out name and age where age is greater than 40, and then walk the people data set, pulling out name and age where age is greater than 50. As an example of two different queries, this is not super interesting. I just copied it off the website. But the actual thing about this is cool, is that this is backed by Hadoop. So this is an extremely powerful small set of lines of code. This says run a parallel query on data sets that could be spread across tens or hundreds of machines and give me back the results. CoreLogic is, this is the classic monkey needs to move a box to the middle of the room problem implemented in CoreLogic. I would encourage you to go download CoreLogic and take a look at it. The work on CoreLogic has been so appreciated in the logic community that the closure community and the scheme community are starting to have a little love in. And the scheme conference next year is actually going to be co-located with the closure conference. And it's been super exciting to have really high powered academic schemers and particularly reasoned schemers, people who are doing logic programming and scheme, sharing ideas and sharing concepts with the closure community. And then here's Datalog in Datomic. So this is the equivalent of SQL, kind of in our world. You can think of expensive chocolate and related product as views. And when you see variables repeated in a query, that causes a join. The thing I like about Datomic Datalog, and I have a personal interest because I helped build it, is that it is just beautiful to read, to do big jobs compared to SQL, right? There's no joining through tables and getting caught in the structure of the join tables and all that sort of muck that you have to do. And all of this stuff is not leaving the language, right? This is not an alternate world that has to be magically brought into the language. This is just people building libraries in the language. Finally, I want to be effective. And I mean the word effective here in the literal sense. I want a program that has effects. This is where functional programming sometimes runs aground. I mentioned before that one of the things people do in functional programming is they sort of wave their hands at this part of it and say, well, you know, you're in Scala, you can say, well, you use functional where it makes sense, and when you need to have an effect, do you have an effect? You do something different. You can take the high road with types and take a monadic approach to effect. Or you can do what closure does, which is to get away from these in place effects by substituting value transitions. So here's where we are today, right? Every little bit of our program is a machine and putting together programs is like assembling things, all of which are moving. So it's no wonder that it's hard to do, right? Composing programs is really difficult because everything is a bunch of moving parts that you're trying to stick together. And this made sense when memory was at 1970s prices. In closure, new memories always use new places. Memory is not that cheap anymore, and we can let garbage collection forget whichever old memories we don't care about. So change is encapsulated by constructors, references refer to a point in time value, and references only see a succession of values. So here we have three values, v1, v2, and v3. We have constructor functions that know how to make values, that maybe take an old value and give you a new one. And then we have atomic succession from one value to another. And then a view of this across time, once upon a time the New York Yankees had this roster, and then somebody got fired and they had that roster, and then two people got hired and they had that roster. Those values are protected by this reference, and observers who are outside that can perceive the identity and remember it and record it, and they can use it in a concurrently safe way. And this is the big one, observers do not have to coordinate. So observers of references or values don't have to coordinate with each other. And people look at this for the first time and go, aren't these just variables? No. They're not just variables. They represent understood semantics of transitions from value to value. If you've ever had to write a lock, then you're doing something different than what we're talking about here. These things do not have to be protected. Here's what it looks like from the API level. I have a reference constructor. In this case, I'm using one called Adam. That makes a reference around the value zero. And then I'm going to take a pure function to do something to the contents of that and some args. And then I have an atomic succession function. So swap says, send a function inside this box, apply the function inside the box, give me back the result. So what will be inside of the Adam after this? It was zero, now it will be 10. And here's mapping that back to the picture. So I have a reference. That reference points to an initial value. I apply a function. We have an atomic succession of the reference, and then it's at a new value. I can do the same thing with, this model is completely general. I can do the same thing with bigger structures. So now the Adam has in it different data. It's actually the equivalent of a Java bean or something. I'm creating a person, and I'm swapping, associating in a different name for that person. I can also have varying semantics. So now instead of having Adam, I have a different kind of ref called a promise, and a different kind of succession function called deliver. So there's not a single semantic for concurrency. There are five or six or seven, depending on how you count them, and they are all safe. This model also scales all the way up to an entire database. So here, I have, this is not closure anymore. This is datomic, which is built in closure. And I'm connecting to a database and transacting against it. Transact is a succession function, just like all of those other succession functions. It's acid, and you start with a value of the database, you get back a new value of the database. In fact, the return value from a transaction is a future. So you can decide whether you want to wait on the return value or not. And inside that future, you have the old value of the database and the new value of the database. It is this atomic succession model writ large. Now, the obvious question, and this came up on the panel, really, you give me back the entire value of the database? What happens if the database is big? What happens if the database is ginormous? And the answer is that these databases are values. So I don't have to give you back, I can evaluate the database lazy. I don't have to give you back the entire database. What can I give you? I can give you the promise that I still know how to get all the rest of it. And then I can fetch the other pieces as needed. So you can grab these slides after. There's a breakdown reference of the various other reference types, but we're not going to look at them today. The last thing I want is for my software to be pervasive. I don't want to have a private island runtime that I have to go and convince managers to install. I want my systems to run on the platforms that people use. And I think those platforms for language like Clojure are the JVM, JavaScript, and.NET. Clojure was originally written for the JVM, that is the most mature implementation, that's where the most work has gone forth. Clojure CLR came next, but Clojure Script has already passed it probably, if not in maturity, at least in widespread use. And the problem, unfortunately, is, and you know this as well as I do, most of you in this room, that languages on the CLR languish until and unless they're blessed by Visual Studio. So I think that Clojure is backwatered on the CLR because there's a strong commitment to using those tools, and the tools are not available. And I would say it would be great if some of you would take it out for a spin and help make Clojure on the CLR more mature and take it in the direction of use that it has seen on Java. So in conclusion, everybody in the industry almost builds things out of transient data structures that don't compose. This includes Java, C-sharp, relational databases, and no SQL. Clojure solves this with persistence and succession of values. Persistence gives us systems that are composable because the parts don't move around while you're not looking. Value succession gives us a way to actually model state that makes sense in that world. There are other choices. You can go down the Monad Road, for example, if you want. These better semantics allow us to write more powerful and more flexible programs, and those semantics are available on real-world platforms where you want to build and run software. I've included at the end of the slides links to a bunch of references. One of these references, and you can easily Google for this. If you Google for Stuart Halloway presentations, I actually have a wiki that's got slides from all the presentations that I do, and I have left, I believe, four minutes for questions. Yes? You mentioned using an outside tool to use the application of the application. Do you have to use a real-world? Yes, so the question is, what is my outlining tool? Yes, I use org mode and snippets. I totally get that that's not where the world is going to go, and I don't want to be an Emacs user. I'm just forced there by its compelling features. But I understand that, you know, telling someone to use Emacs, if you're not already an Emacs user, did you just tell me to F myself? Right? Because it's a hard learning curve. There's IDE support for closure in all of the mainstream Java IDEs, and it has characteristics that are more familiar and friendly to developers that don't matter very much to me, and what it's missing is general text editing capability, speed, and in particular, a structural editing mode called par edit. So if you've never worked in a language that had a structural editing mode, I would encourage you to do a Google search for par edit, P-A-R-E-D-I-T. That's the big reason that most closure users stick with Emacs once they get hooked on it. Other questions? Those of you who are going to be watching this on camera, I'm in a brightly sunlit room, shading my eyes so I can see people. Yes? So the question is, is it possible, given that the data structures evolve by creating new immutable data structures, can you get back to the old ones? The answer is you could if you wanted to, and it depends. So closure does not, by default, keep the last ten versions of a ref and let you go back to them, but the ref semantics, the model of refs, you could write a ref that had those semantics if it made sense to do in your program. Also, obviously, if you hold on to a value, if you go back and look at my picture of the observers, some of them are looking at old value, as long as you're still looking at it, you're still looking at it. Now, on the durable side of persistent data structures, when you go to Datomic, that works a little bit differently because in Datomic, everything is time-stamped and nothing is ever thrown away in that sense. So in Datomic, you can actually put a filter on the database that shears off data from some time range. So you can say dot as of a month ago, teleport your database back in time a month and query against it there. That's the kind, and it's a great example of the kind of thing that's theoretically made possible by this. Not all the reference types deliver that capability, but it's possible. One last question. Okay. Right, so how does this data structure copying thing work efficiently? So if you imagine a linked list, if I have a linked list with four guys on it and you stick a guy on the front, I don't know you did that, and that's called shared structure. This guy's talking to that guy and nobody cares. The data structures in Datomic are all listed in sideways. They're all trees, and they have the same property. So you have an associative data structure over here that's represented as a tree. You make a change to one thing. You get a new tree that's pointing willy-nilly all into the old tree to reach all the old stuff except the one. So the cost you pay is what's called a path copy, and in closure, the paths go five deep. So it's actually, and they are, it's log 32 in instead of constant time, but given that it only goes five deep, it falls out in the wash. In Datomic, on the durable side, those trees are three deep. So they're incredibly wide branching trees, so you don't pay a big penalty for that. That is slower. Those data structure updates are slower than a purely mutable data structure. So on the flip side, you get the speed back on reads, right, because you don't have to coordinate, you don't have to lock. And so if you live in a world where there will be more multi-core processors in the future, closure will become better and betterer. And if you live in a world where multi-core processors are going to be defunct, then the stuff I'm talking about is ridiculous and you should ignore me. So my name is Stuart Halloway. Make sure you remember to fill out eval cards. I have discount coupons for this O'Reilly closure training video and closure stickers. If anybody didn't get those, come and see me after the talk. Thank you very much.
important single example of a more general changes via succession of values. So you have an systems. I want to be power or knowledgeable. And to be knowledgeable, I actually have to have position. And persistent data structures are a A lot of functional languages support this, so with one data structure than to have 10 functions should be careful about the use of the word programming doing higher order functions. This programmers don't encourage you to access the private. The place where you will see that syntax metal, I mean access to the platform. I mean no need to drop down to JavaScript. And so he put Clojure as simple, wrapper-free. And this is a knowledgeable and powerful. I want to talk now to work in a dynamic language. So Clojure is a this higher order functional style of programming what actors are in more than one? So here's how structure has in it more than names and I just Hadoop. So this is an extremely powerful small we don't care about. So change is encapsulated by value, and references only see a succession of level. I have a reference constructor. In this database. It is this atomic succession model writ written for the JVM, that is the most mature spin and help make Clojure on the CLR more mature tool to use the application of the application. you've never worked in a language that had a wanted to, and it depends. So closure does not,