The F#orce Awakens
Gabasova, Evelina
Let's dive together into the the world of Star Wars! We'll use the force of F# and R to process publicly available datasets relating to the Star Wars movies to find out who's the most important character in the stories and why were the prequels so unsuccessful. On the way, you'll see why F# is a great language for data science - from preprocessing the data to visualizing them - and you'll also learn how you can use similar data processing pipelines to get interesting insights from your own data.
 Okay, I think it's time to start. So welcome to the Force Awakens with F-Sharp. I don't even know how to pronounce this correctly. So I'm Ewerina Gabasova and I work as a postdoc researcher at Cambridge University in cancer research. So I deal a lot with DNA mutations, things like that, and that's an incredibly complex type of data. And when you look at it, it's really hard to understand what you are doing, and even if you fit some kind of statistical machine learning model, it's really hard to see what's actually happening there. So this is an example of metabolic networks, and you really can't see anything. I can't see anything because I'm not a biologist. So in my free time, I like to play with other data. I suspect they won't be using this title for the actual episode 8. So let's talk about Star Wars. Who likes Star Wars? Yeah, thanks for coming. You are in the right stalk right now. So some time ago, I decided, well, let's analyze Star Wars. I actually started doing this right before the premiere of episode 7, so there was a lot of hype around Star Wars. So I was thinking, so what kind of data can I actually analyze? And well, there are obviously the movies, but it's kind of hard to analyze movies, right? So the next data set that's available about Star Wars are actually the screenplays or scripts. And there are multiple different repositories online where you can go and download scripts of almost any movie you are interested in. And they publish it very quickly. So for example, the script for episode 7 was there, I think, in January. I'm not really sure what's the legal status of that, but if you want to look at it, you can. So I looked at the scripts. And this is an example of a script from episode 4, the original Star Wars. And the great thing about this is that they have a very standard type of format. So you can probably see that there is some title of a scene. And they always start with int or x, meaning interior or exterior. Then there is some description of a scene. And there is a name of people speaking and what they are saying. And because we are programmers, we know that this is actually quite easy to parse. Or if it has this type of structure, we can parse it. So I decided, well, let's look at interactions in Star Wars. And maybe I can even extract social network by looking at who speaks with whom in the scenes. So I went on and downloaded all the script files of all the 7 movies right now. I don't know if I will continue with this in the future, but I have 7 of them now. And I looked at them. And because they are published online, they are usually in HTML. And this is how it looks. And that's also pretty nice, right? There is a pre-formatted HTML block with the scene name in bold. And then there is a description and again, name in bold and in capitals. That's easy to parse. Well, usually. So when I first started doing this, I ended up with a bunch of rejects. Because I wanted to match all the different types of how a name can appear in a script. But because I'm working in F sharp, I would like to show you another way how to parse these things. I know if you have been to Scott Vlaschens talk before me, he was talking about parsers, I'm not doing anything that fancy. But when I looked at the structure of the script, it's actually quite simple. But I, well, this is just an example of the standard format of the script, how it looked. But I want to really write something that I can read. I don't want to deal with how it looks underneath. So what I used something that's called active patterns in F sharp. And there you can write something like this. You just parse a screen. You split the script into elements. And then you can just match each element here with a scene title or a name. And that's very readable. And you don't even have to know what's happening underneath. But I'll show you what's happening underneath. This is something called the active pattern in F sharp. So when you want to match something against some rejects, you don't have to put the rejects into a function that's actually doing the matching. You can just look at, you can define something like this. This is called the banana clips operators. So you can say that a piece of text is either a scene title or a name or something that I'm not interested in. And you can hide all the ugly rejects in here. And by the way, the rejects for names is very complex because in Star Wars, like characters are named anything. You don't get people named C3PO in real life. So this had to be quite complex because it had to match all sorts of dashes, slashes, everything. And I don't want to care about this. For me, rejects are something that it's right only. I don't want to look at it again. So I can hide it in this kind of thing. So what this does is it takes the string as an input and then it matches it against each of the rejects and returns this kind of thing, like scene title and text or the name text or word. And then when I do pattern matching over it, I don't have to care about what it does underneath. And this is readable. When I went back to my original implementation with all the rejects all over the place, I had no idea what was happening there. So for me, this is the way to write parsers in Fsharp or at least for very simple cases. And if you want to play with Fsharp, definitely look at active patterns because they allow you to hide the implementation details. And then you don't have to care about it in the more high level functions. So I thought, now I'm set. It wasn't the case. So I tried to run it on all my seven different script screenplays. And yeah, it was a trap. Because some of the files have completely different structure. For example, the names are not in bold, are not centered. They are at the beginning of a line followed by colon, et cetera. So I had to write another parser for this type of screenplays and it was much more work as usual. And that's a general case in anything data related. You spent 90% of your time just cleaning up the data. And now this also still has some kind of structure because all the names are in capitals at least. You would think so. For example, this one, by the way, the one made on is this scary guy. But if you look at the second letter, it's not an I. It's a lowercase l. I don't know if it came from some OCR system and it just decided that I is a lowercase l. But I, and for things like these, I actually had to put in explicit modifications because I didn't find a way how to deal with this systematically. And that's every time you have to deal with data. So I went through this and I thought, okay, now I'm all set. I have all the characters and all the scenes. It's all nice and easy from now on. Well, no. Because some characters don't even speak in Star Wars screenplays. They are mentioned there but they don't speak at all. For example, R2-D2. If you actually go into the script, they say things like these, like R2 frantically beeps something. Well, thanks. Or Chewbacca doesn't speak at all. So this is Chewbacca barks a comment and Han replies, boy, you said it, Chewy. No, you didn't. So actually, Peter Mayhew who plays Chewbacca, I think he did a very good job playing him because with this type of instruction in the script, what can you do? So I decided, well, when I extracted all the names in all the scenes, I decided, well, I actually have to include these characters because they can't be left out, right? You can't just have Star Wars social network or something without R2-D2 or Chewbacca. So I went on and actually extracted also all the mentions of all the characters in screenplays and tried to estimate how many times would they have spoken if they have spoken there. So I came up with very, very scientific equations. Well, you don't have to care about this, really. It's basically just scaling the number of times a character is mentioned by the times they actually speak. So on average, actually, every character is mentioned about two times as much as they speak explicitly in script. So I just counted how many times, let's say, Chewbacca is mentioned and waited it by how many times Han is mentioned and tried to compute how many times Chewbacca would have spoken in the script. So everything is good now, right? Well, sorry I didn't include some of the characters because there are actually so many characters that don't say a word that Ewoks didn't make it, for example. So the only ones that I included manually in this way were R2D2, Chewbacca, and VV8 from Force Awakens. And then when I look at the characters that my algorithm extracted from all the screenplays, I found out, well, actually, some of the other characters, they appear there but they don't say a word in some of the episodes. So they messed up my analysis again. And if you have seen episode 7, you probably know who I'm talking about. So now I decided, well, I should somehow check if all the characters, because there was such a mess, actually appear in the screenplays. And if there are actual characters, if it's not just some artifact of my algorithm. And I propose a theorem that there is an API for everything. And there is an API for Star Wars. You can go to swapi, StarWarsAPI.co, and you can find an API for Star Wars. So I looked at the website, and it's quite nice, quite well documented. You have all sorts of example requests and what it returns. And you can get information on characters, on starships, on planets, on vehicles, I think as well, everything. And they have a lot of wrapper libraries. So when I looked at them, there was C sharp, Python, R, Java, Go, Ruby. There was no F sharp, but well, maybe I can just use one of the other ones. But then no, no, no. Well, I'm doing this in F sharp. Let's do it properly. And then I looked at, for example, this is the C sharp one. And you can see that the code is not very difficult. Well, these are all just get and set methods and things like that. So there is not actually much going on. And this is just an example of one of the wrappers around, I think, people. And you can see it's 150 lines, and most of them don't actually do anything. So I wanted to show you how we can do something like this in F sharp. And if you have seen any of the F sharp talks, some of them mentioned type providers. So this talk is partly me saying I lost type providers. If you are doing anything with data, type providers are amazing. So I wanted to show you how I can use type providers to write something like all the wrappers that were there in all the different languages in a very small amount of code. So I'll just load the F sharp data library, which contains some of the type providers. And now this is actually just an example request to the Star Wars API. And what I get from this is some document that basically just answers the request. And what I will do is I will just, I don't have to read the documentation. I don't have to do anything. I can just take this and create a type provider. And it will be a type provider for a Star Wars person. And it will be a JSON provider. And I will give it the URL. Create this and I have basically everything I need right now. So now I can write a function to get me information about any person in Star Wars. So get person. And I will take an ID as a parameter. And now I will just call person load. And I will give it the request that I want. And I will give it the ID. I need normal brackets here. And that's it basically. Now I can have a look at the first person. I need to change this into string. And now I get the information on the first person. And again, it returns a JSON, which is not very readable. So I can just type p. and get all the information here in my intelligence, including the types of all the elements. So for example, I can look at who the first person actually is. So I will look at name. I know that it's a string. And it's Luke Skywalker. Do you know what's the, let's say, his mass? How much he weights? Did you ever want to know something like this? Well, he weights 77 kilos. A lot of very important information. But what I wanted to show you particularly is that basically in these three lines, I have everything that was there in the full big wrapper. And I don't have to care about it at all. I don't have to read any documentation, write any methods. This is all I need. So I actually went on and wrote a full wrapper around the API. And you can see it's not very imaginative code either. It's just defining the type providers and defining some of the functions. And that's it. That's everything. And on the same day that I discovered the Star Wars API, I actually just went back and sent a pull request and now there is an F sharp wrapper. Thank you. And in case you were wondering other random things about Star Wars, I can show you some more information. For example, do you know what's the most common eye color in Star Wars characters? You always wanted to know this, right? So the first most common eye color is brown. So for example, Yoda has brown eyes. Who knew? The second most color is blue. And the third most common color is yellow. So Darth Vader has yellow eyes, all the important information. And you can also see, for example, in episode four, the original Star Wars, Luke Skywalker once walks into a prison cell and Princess Leia says, aren't you a bit short for a storm trooper? So we can see, so is actually Luke Skywalker a bit shorter than average Star Wars character or not? Wow. It's very easy. I just get his height, which is 172 centimeters. And I also downloaded information on all the other characters. And it's 174 centimeters on average. So yes, Luke Skywalker is shorter than average. So she was right. So all the important information that you always wondered about, right? So well, let's move on. So if you want to look at it, it's on my GitHub and you can get to it from the Swarovie website as well. And play with it. It's a lot of fun information. So now what I had right now, when I ran all the code that I was describing, I had characters, I had the scenes, and I knew that all the characters are actually appearing in the films. Then Star Wars was bought by Disney and then I came across this analysis and they actually analyzed 2,000 screenplays and they were looking at how much women characters speak and how much male characters speak. And they were just comparing scenes and how much screen times they get. And they specifically looked at Disney. And it was sort of depressing because in almost all the Disney movies, men speak much more than women, etc., even if it's about princesses and things like that. And because I had almost the exact same data, I thought, well, maybe I can replicate this using my Star Wars dataset. And again, type providers. Because what they did in this analysis is that they went onto IMDB, extracted a list of actors playing in the individual films, and then they looked at them if they are male or female. And I can do exactly the same thing with my type providers. Because there is not only a type provider for Jason, there is a type provider for HTML as well. So IMDB, HTML type provider now. So let's just open everything again. And as you can see, I'm also giving it just the URL of some page describing a film. And this is actually from episode 7, so I will load it now. And let's... I don't even have to look at the website. Now I can do episode 7. And I know that there are some lists and some tables. So I will look at the tables. And the tables are these. I don't have to go through the website at all. I see everything here in F-sharp. And this is all the code I really need to access it. So I can look at the cast. And they have the cast in credits order verified as complete. And here I have a nice printing function to actually look at it properly. So these are all the people that play something in episode 7. And for example, did you know that Daniel Craig plays a stormtrooper? He's somewhere here. Yeah, Daniel Craig plays a stormtrooper and he's uncredited. So this gives me quite a lot of information. And again, it's in my Rappel. I don't actually have to look at the website and go through it. So with a bit more code, I put together this graph. So this is comparing episode 1 to 7 based on what's the percentage of scenes where, what's the percentage of dialogues with men and what's the percentage of dialogues with women. And actually, sorry, I put women and robots together because they are like other genders. Because otherwise, it would be even worse. So episode 7 indeed has more women speaking, but still almost 70% of the dialogue is men. Still better. The worst one is actually episode 4. But enough of this. I had right now, as I said, all the characters and their relationships and where they speak. So I decided to put together a social network by putting together characters that speak in the same scene. If they do, they are connected by a link. And I can visualize this. And because, well, what do you do if you want to visualize something nice and put it on the web? You use JavaScript. So I went for D3JS, which is an amazingly powerful library in JavaScript for visualization. You can write anything there. The downside is that you can write anything there. Because it's so powerful that you can't really learn it. You have to go online and look at various examples of what people did with it before so that you can copy it and put your own data into it. Anyway, I don't really want to use JavaScript. And as I said, I want to do everything in Fsharp here. So there is this new project called Fable. So it's using Babel. This is the current logo. And that allows you to write code in Fsharp and translate it into JavaScript. I think it's called transpiling. Is that the correct term? So it's a transpiler. And it's actually really neat. It's a very new project. I think it started about six months ago. And here on the right-hand side, you can see the code in JavaScript from an example on D3JS that I just downloaded from the web. And you can see that most of the code on the left-hand side, it's the code in Fsharp. That's actually calling the D3JS library. And you can see that it's very similar, actually. Right? Well, instead of var, you have let. But otherwise, all the function calls are very similar. Sometimes, you have to deal a bit with types, but it's not very painful. And it allows you to basically just translate your JavaScript code into Fsharp. And you can call any functions from Fsharp and have all the type safety and everything. And I just wanted to show you also the translated code. So this is how it translates the code into JavaScript. And you can see it, for example, this is calling something called the first layout of networks. And it's actually very readable. Like there are no thousands of underscores everywhere. And you can go there and see what's happening. And that makes any debugging so much easier. So I actually went on and did all the things in Fable. And if you want to play with it, there are many examples online by playing games from Fsharp, translating it into JavaScript. There are also some examples with Node.js. And it's really nice, really. So let's go to the actual social networks that I promised. So this is the social network of all the Star Wars movies put together. And because I promised JavaScript, here it is. So this is an interactive visualization of Star Wars movies. Let's make it a bit bigger. So you can see that the big black note is actually Darth Vader. And I guess you can already see some patterns emerging. Whenever you are working with data, try to visualize them because that gives you so much information. So I guess right now you can probably guess that on the left-hand side are the new prequel episodes. And you can see that the social network is a mess. You can see that there are so many nodes and it's very dense. Then in the center are the original episodes. And on the right-hand side is the episode 7. We can even look at the episodes individually. So this is the first episode of the prequels. And you can see it's quite densely connected. It has several main characters. I tried to color them. So this is, for example, Qui-Gon. This is Anakin. This is Jha-Jha. And when we compare it with the episode 4, the original Star Wars, you can probably immediately see some difference. The network is much sparser. It has quite a few, only a few major characters, and they are connected to each other. And if you compare it with Force Awakens, the social network is bigger. But there are not that many characters as in the prequels. So that already tells us something about the structure of the story and how understandable it is probably. And we go even further, we can quantitatively compare the different episodes. So the first thing that you can think of is how large networks are. So this is just a graph showing a number of characters. And these are only the characters that speak at least in two scenes. And only the characters that are explicitly named. So if someone is just a Stormtrooper, I didn't include them there. And yeah, episode 7 has the most characters. And then the original episodes have fewer, about 20 main characters. And then in episode 7, it jumps up again up to 27. So let's hope they don't continue with this trend. Otherwise, they will get to the episode 1 territory, and you probably don't want to go there. And then there are various scientific methods to compare networks. And the first one is called density, which sounds fancy like a density of a network. But what it really does, it just tries to compare the number of connections that are in the network with the number of connections that could be potentially there. So if you want an equation, this is it, just divide the number of existing connections in the network by the number of total connections that could be there. And when we look at this, actually, the episode 1 and 2 have the lowest density. Because they have a lot of characters that are only vaguely connected, and they have all these debates in the galactic senate that are not very interesting. And actually, interestingly, episode 6 has quite a high density, and that's maybe because I didn't include the Ewoks in there. And episode 7 actually has about the same density as the original episodes, and so does episode 3. And actually, if you look at IMDB, you will see that episode 3 has the highest rating out of the prequels. So maybe density has something to do with the quality of the story as well. And you can look at other measures. One of them is the clustering coefficient, which tells us how locally connected the network is and how much the characters actually speak to each other. So if you look at this green guy, he has three neighbors. And you can see if the three neighbors are all connected to each other as well or not. So for this one, there are two of them connected. For example, for this one, all his friends are connected. So that tells you how connected your networks are. And in terms of the story, this basically means if the story is following one character that just interacts with other people and the other people don't talk to each other, then that will have a very small clustering coefficient. And if the story follows a group of people that talk together and interact with each other a lot, then it will have a large clustering coefficient. And this is the equation. We don't actually have to care about it very much. I don't want to go into details. But if I plot it for all the different episodes, again, episode one has the lowest clustering coefficient because there are all these weird characters that are not really talk very much. And what's nice is that episode seven has about the same clustering coefficient as the original episodes. And actually, episode three here has quite a small clustering coefficient as well, which is interesting. It tells us something about the story. And I'm not claiming that small clustering coefficient means a bad story. I think it makes sense in something. For example, if the main hero is going through some obstacles and meeting other characters that are helping him on the way but don't really interact with each other, that can be a good story. But I think in Star Wars, it actually tells us something about maybe how the network is structured. And actually, I think in this case, it roughly correlates with the quality as well. And then we can look at local characteristics in the network. So the first one, the most basic one is degree. And that's just by looking at how many connections a character has in the network. So for example, this guy, he's very important because he has six connections in this network, a small one. But this guy is less important because he has just three connections. And that tells us something about centrality of the characters in the social network. So this just represents how many characters each one of them speaks to. And yeah, it's just basically number of links that are outgoing or incoming into a note. And I want to compare it with another measure of centrality in a network, which is called betweenness. Because there are many measures of seeing who's the most central in a network. And the degree just tells us if someone talks to a lot of other characters. But betweenness tells us how important a note is for communication in the network. So I will explain again. So if we look at this guy and these two other guys, the only way they can communicate with each other is through the green guy because they don't know each other directly. And these guys, on the other hand, if they want to talk to each other, only one half of the communication would go through the green guy because they know each other through some other person as well. So I can, for example, ask, so if Princess Leia wanted to talk to Jar Jar Binks, who would she have to go through to pass a message to him? So that tells me how important a character is within the story. Because some characters may be important just in one part of the network, but if a character speaks to a lot of different characters across the whole episode, then that means he's probably more important to the story. So again, this is the equation. And now it's a bit more complicated because for each pair of notes, we look at the number of shortest paths between them and then look at how many of these shortest paths go through the specific node. And we sum it over all different pairs of nodes in the network. And it's already quite hard to compute, right? Because you have to compute all the shortest paths between all the nodes in the network and then look at how many of them pass through the node that you are interested in. And if you are doing anything with data, there is always a package for it in R. So there is a library in R called iGraph. And if you are interested in betweenness, there is a function called betweenness. And if you are, as I was working in Fsharp, you can do something like this. Just call iProvider, RProvider.iGraph, and call the betweenness function from R. And again, you get intelligence and everything. And this makes anything like this very easy because you do all the heavy preprocessing in language that you are more comfortable with or that's a bit more safe than R. And call the algorithms that are already implemented there. And I actually do this quite a lot because R is a nice programming language, but just for data science. If you want to do any general programming, then it can get really, really painful. And in Fsharp, you get all the safety of type inference. So it really helped me when I was writing my parser, for example. And I just knew all the heavy data science in R and I can pass data to it directly from Fsharp. So let's look at the centrality. So who is the most central in episode seven, let's say? So actually the person who's having the most connections is Spoh. He's one of the resistance pilots. And that's because he talks to a lot of people across the whole network. And he talks to all the other resistance pilots as well. And there are quite a lot of them. And the most second central is Finn because he also talks to a lot of other people. And then Han, Chewbacca, and VV8. And for VV8, this is just an estimated number because I didn't find him in the screen place as I was explaining. So does this tell you something about who's actually important in the story? Sort of. But you might be missing some of the main characters in this. So this is actually a bit weanness. And now you can see that suddenly Kylo Ren and Ray jumped up a lot. And that's because they are more important to the actual, they are more important by connecting different communities within the network. So for example, Kylo Ren is very important because he is one of the few people that talked to Han and Chewbacca and Ray, but he also talked to Snoke, which puts him into the center of the network a bit more. But also Poe talked to him, which already gave him quite a boost in betweenness as well. So this tells you a bit more about who's important in the actual network. And if now you might be asking, so who's the most important across the whole Star Wars, right? So before the new episode, this is just Poe celebrating that he's the most central. So who is the most central overall across the whole Star Wars universe? Before episode seven, it was this guy because he was talking to quite a lot of people in the prequels as well as in the original trilogy, a little, although he appeared physically in only the first one of them. And also what I did in this analysis when I got this guy, it was because I just looked at the names in the screenplays. And actually they are making a very big distinction between Anakin and Darth Vader because they never appear in the same scene, so they appear like two completely different characters. And when I added the new episode and also merged Anakin and Darth Vader, still Darth Vader is ruling the galaxy, he is the most important one. Everyone else is just lying there dead. So also I made a quick sample with Neo4j. So if you want to learn something about Neo4j, I have a link at the end. You can go there and play with it because it's a very nice data set and you can just extract connections between people. And what's important here is that you actually understand what's going on because all the people have names that you recognize. And that makes any learning so much easier than if you have just some anonymous customers or something like that. You can understand what's happening. So this is for example just looking at characters that play in the same movie and how they are connected to each other and computing the degree. Yeah, degree just within a single movie. And you can see that Neo4j is actually quite readable because here I'm just looking at names of characters that appear in... Yeah, I'm looking at episode four, New Hope, and I'm looking at characters that talk to other characters that appear also in New Hope and looking at the number of scenes they spoke together with in. So this is how you would do it in Neo4j and you can play with it. And it's not just completely a toy example. I did something similar by analyzing Twitter and I actually analyzed the social network around F-Sharp, the F-Sharp Software Foundation, which is the home of open source F-Sharp. And it has this Twitter handle, F-SharpOrg. And I looked at all the other Twitter handles that are connected to it. And then I was looking for the most central people there. And actually this is the order based on degree. You can see that the first one is Don Sime, who is actually the author of F-Sharp. So if you are an alien and come to Twitter and you say, oh, so who is the most important in C-Sharp, you can do something similar. I don't know who you get. But it tells you something about the actual network and some meaning about it. And the second one, it's, I think it's the official Microsoft side of F-Sharp. The fifth one is the official side of the community of F-Sharp. And the fourth one is Thomas, who's speaking tomorrow, if I'm correct, in the morning. So if you want to see some important people in F-Sharp, Twitter says you should go see him. And these are information from November 2014. So it might have changed. But the problem with analyzing data from Twitter is that it takes an awful long time to download because there is all the rate limiting. So I didn't replicate it afterwards. But I might do that because, well, it tells you something about how people communicate with each other and what's happening. So if there are any changes, I can see it probably. And also another example, well, this is quite a famous example in network science. And it was a company in Hungary where they had two factories and some headquarters. And they were having problems because there are always these rumors spreading in the factories and they had no idea why and what's happening because the headquarters, they were just issuing orders and issuing messages. But there are always rumors and they were not true. So they actually called, well, in these olden days, they called social scientists who talked to people and said, so who do you talk to if you want to get more information about anything? And they named a few people. And this is the social network that they constructed from it. And what's interesting is that, well, you can probably see that I think the pink one is the headquarters. The other one are the factories. And the most important note is here. It's not in the headquarters. And that was because there was this guy who was doing health and safety. And he was actually traveling around the factories and talking to everyone. So he was the most important person in the company for communication. So now they knew that if they want to spread some kind of information, they have to talk to him because he will spread the correct information to the others. And what you would do now, you wouldn't get social scientists to talk to people. You would just explore the emails that are sent within your company or Slack messages, like who replies to whom, things like that. And because you have access to all this information, you can actually do that. And then you can look at these simple measures, like betweenness and degree centrality, things like that. And you immediately get some information that might be actually useful for you. So as I mentioned, you can do social network analysis. And it's actually a lot of fun. And you can look at how you communicate on Slack, who sends emails to whom. And you can also analyze supply grades. If there is a blackout in some part of your network, what parts will get affected as well. So network science is actually very important. And at the end here, I have biological networks. And that's actually the area where I work because I look at how genes interact with each other, what protein binds to what region of DNA. And then I can see what's happening, how they are interacting with each other. And this is really not just a toy example, because I was reading a paper the other day and they were actually looking at the betweenness. And I was like, ah, I know what betweenness is. I did it in Star Wars. And here they were claiming that genes that are important in cancer, that are tumor suppressors or oncogenes, that they have higher betweenness in the biological networks. And because I was analyzing the Star Wars networks before, I knew almost exactly what it means in terms of the actual network, because I got a feeling for it by looking at very fun data set. So I want to encourage you to actually play with data, because you get more of it than just some fun fights about Star Wars. And right now I went through quite a lot of things. I went through script parsing in a functional way. You saw the active patterns. I was calling R and JavaScript from F-sharp. I think that's actually the future of data science. Just call everything from everything, because there are different tools in different languages. I was showing type providers. I showed you the HTML type provider and JSON type provider. And there are many more. There is one for SQL, there is one for... I can't even think of all of them. I use the CSV one, because CSV is the format for data science. And I would really like to encourage you. If you are interested in data science, if you have seen some of the talks on R, for example, then you might be thinking, well, maybe I should go into data science or start with fun data sets. Because by analyzing them, you know what's happening there. And you get insights when you actually see some of these algorithms applied in real world. And this is, I think, a great way to learn data science. If you want to know more about F-sharp, I put some links there. I have the slides online already. And these are some of the Star Wars resources that I put together. So, yeah, you can read all the scripts, even if you are not interested in any quantitative analysis. You can play with the Star Wars API. You get all sorts of very important information. And all the information that I was showing you is actually on my GitHub. And you can even play with the social networks. And I saw some people doing actual social network data science, playing with it and trying to, for example, overlay the different episodes against each other to see who corresponds to whom in the social network in episode seven, et cetera. And I have some blog posts about this. And as I mentioned, I have a Neo4j demo. And, yeah, play with data. And this is actually the website where I put the slides. So go to eval.inghg.com slash Star Wars talk. And they are all there. And thank you. Are there any questions? So, any questions? I can't actually see you properly. I can't see anything. Well, if there are no questions, then it's time for lunch, I guess. Thanks, guys. you
 Okay, I think it's time to start.  So welcome to the Force Awakens with F-Sharp.  I don't even know how to pronounce this correctly.  So I'm Ewerina Gabasova and I work as a postdoc researcher at Cambridge University in cancer  research.  So I deal a lot with DNA mutations, things like that, and that's an incredibly complex  type of data.  And when you look at it, it's really hard to understand what you are doing, and even  if you fit some kind of statistical machine learning model, it's really hard to see what's  actually happening there.  So this is an example of metabolic networks, and you really can't see anything.  I can't see anything because I'm not a biologist.  So in my free time, I like to play with other data.  I suspect they won't be using this title for the actual episode 8.  So let's talk about Star Wars.  Who likes Star Wars?  Yeah, thanks for coming.  You are in the right stalk right now.  So some time ago, I decided, well, let's analyze Star Wars.  I actually started doing this right before the premiere of episode 7, so there was a lot  of hype around Star Wars.  So I was thinking, so what kind of data can I actually analyze?  And well, there are obviously the movies, but it's kind of hard to analyze movies, right?  So the next data set that's available about Star Wars are actually the screenplays or  scripts.  And there are multiple different repositories online where you can go and download scripts  of almost any movie you are interested in.  And they publish it very quickly.  So for example, the script for episode 7 was there, I think, in January.  I'm not really sure what's the legal status of that, but if you want to look at it, you  can.  So I looked at the scripts.  And this is an example of a script from episode 4, the original Star Wars.  And the great thing about this is that they have a very standard type of format.  So you can probably see that there is some title of a scene.  And they always start with int or x, meaning interior or exterior.  Then there is some description of a scene.  And there is a name of people speaking and what they are saying.  And because we are programmers, we know that this is actually quite easy to parse.  Or if it has this type of structure, we can parse it.  So I decided, well, let's look at interactions in Star Wars.  And maybe I can even extract social network by looking at who speaks with whom in the  scenes.  So I went on and downloaded all the script files of all the 7 movies right now.  I don't know if I will continue with this in the future, but I have 7 of them now.  And I looked at them.  And because they are published online, they are usually in HTML.  And this is how it looks.  And that's also pretty nice, right?  There is a pre-formatted HTML block with the scene name in bold.  And then there is a description and again, name in bold and in capitals.  That's easy to parse.  Well, usually.  So when I first started doing this, I ended up with a bunch of rejects.  Because I wanted to match all the different types of how a name can appear in a script.  But because I'm working in F sharp, I would like to show you another way how to parse  these things.  I know if you have been to Scott Vlaschens talk before me, he was talking about parsers,  I'm not doing anything that fancy.  But when I looked at the structure of the script, it's actually quite simple.  But I, well, this is just an example of the standard format of the script, how it looked.  But I want to really write something that I can read.  I don't want to deal with how it looks underneath.  So what I used something that's called active patterns in F sharp.  And there you can write something like this.  You just parse a screen.  You split the script into elements.  And then you can just match each element here with a scene title or a name.  And that's very readable.  And you don't even have to know what's happening underneath.  But I'll show you what's happening underneath.  This is something called the active pattern in F sharp.  So when you want to match something against some rejects, you don't have to put the rejects  into a function that's actually doing the matching.  You can just look at, you can define something like this.  This is called the banana clips operators.  So you can say that a piece of text is either a scene title or a name or something that  I'm not interested in.  And you can hide all the ugly rejects in here.  And by the way, the rejects for names is very complex because in Star Wars, like characters  are named anything.  You don't get people named C3PO in real life.  So this had to be quite complex because it had to match all sorts of dashes, slashes,  everything.  And I don't want to care about this.  For me, rejects are something that it's right only.  I don't want to look at it again.  So I can hide it in this kind of thing.  So what this does is it takes the string as an input and then it matches it against each  of the rejects and returns this kind of thing, like scene title and text or the name text  or word.  And then when I do pattern matching over it, I don't have to care about what it does underneath.  And this is readable.  When I went back to my original implementation with all the rejects all over the place, I  had no idea what was happening there.  So for me, this is the way to write parsers in Fsharp or at least for very simple cases.  And if you want to play with Fsharp, definitely look at active patterns because they allow  you to hide the implementation details.  And then you don't have to care about it in the more high level functions.  So I thought, now I'm set.  It wasn't the case.  So I tried to run it on all my seven different script screenplays.  And yeah, it was a trap.  Because some of the files have completely different structure.  For example, the names are not in bold, are not centered.  They are at the beginning of a line followed by colon, et cetera.  So I had to write another parser for this type of screenplays and it was much more work  as usual.  And that's a general case in anything data related.  You spent 90% of your time just cleaning up the data.  And now this also still has some kind of structure because all the names are in capitals at least.  You would think so.  For example, this one, by the way, the one made on is this scary guy.  But if you look at the second letter, it's not an I.  It's a lowercase l.  I don't know if it came from some OCR system and it just decided that I is a lowercase  l.  But I, and for things like these, I actually had to put in explicit modifications because  I didn't find a way how to deal with this systematically.  And that's every time you have to deal with data.  So I went through this and I thought, okay, now I'm all set.  I have all the characters and all the scenes.  It's all nice and easy from now on.  Well, no.  Because some characters don't even speak in Star Wars screenplays.  They are mentioned there but they don't speak at all.  For example, R2-D2.  If you actually go into the script, they say things like these, like R2 frantically beeps  something.  Well, thanks.  Or Chewbacca doesn't speak at all.  So this is Chewbacca barks a comment and Han replies, boy, you said it, Chewy.  No, you didn't.  So actually, Peter Mayhew who plays Chewbacca, I think he did a very good job playing him  because with this type of instruction in the script, what can you do?  So I decided, well, when I extracted all the names in all the scenes, I decided, well,  I actually have to include these characters because they can't be left out, right?  You can't just have Star Wars social network or something without R2-D2 or Chewbacca.  So I went on and actually extracted also all the mentions of all the characters in screenplays  and tried to estimate how many times would they have spoken if they have spoken there.  So I came up with very, very scientific equations.  Well, you don't have to care about this, really.  It's basically just scaling the number of times a character is mentioned by the times  they actually speak.  So on average, actually, every character is mentioned about two times as much as they  speak explicitly in script.  So I just counted how many times, let's say, Chewbacca is mentioned and waited it by how  many times Han is mentioned and tried to compute how many times Chewbacca would have spoken  in the script.  So everything is good now, right?  Well, sorry I didn't include some of the characters because there are actually so many characters  that don't say a word that Ewoks didn't make it, for example.  So the only ones that I included manually in this way were R2D2, Chewbacca, and VV8 from  Force Awakens.  And then when I look at the characters that my algorithm extracted from all the screenplays,  I found out, well, actually, some of the other characters, they appear there but they don't  say a word in some of the episodes.  So they messed up my analysis again.  And if you have seen episode 7, you probably know who I'm talking about.  So now I decided, well, I should somehow check if all the characters, because there was such  a mess, actually appear in the screenplays.  And if there are actual characters, if it's not just some artifact of my algorithm.  And I propose a theorem that there is an API for everything.  And there is an API for Star Wars.  You can go to swapi, StarWarsAPI.co, and you can find an API for Star Wars.  So I looked at the website, and it's quite nice, quite well documented.  You have all sorts of example requests and what it returns.  And you can get information on characters, on starships, on planets, on vehicles, I think  as well, everything.  And they have a lot of wrapper libraries.  So when I looked at them, there was C sharp, Python, R, Java, Go, Ruby.  There was no F sharp, but well, maybe I can just use one of the other ones.  But then no, no, no.  Well, I'm doing this in F sharp.  Let's do it properly.  And then I looked at, for example, this is the C sharp one.  And you can see that the code is not very difficult.  Well, these are all just get and set methods and things like that.  So there is not actually much going on.  And this is just an example of one of the wrappers around, I think, people.  And you can see it's 150 lines, and most of them don't actually do anything.  So I wanted to show you how we can do something like this in F sharp.  And if you have seen any of the F sharp talks, some of them mentioned type providers.  So this talk is partly me saying I lost type providers.  If you are doing anything with data, type providers are amazing.  So I wanted to show you how I can use type providers to write something like all the  wrappers that were there in all the different languages in a very small amount of code.  So I'll just load the F sharp data library, which contains some of the type providers.  And now this is actually just an example request to the Star Wars API.  And what I get from this is some document that basically just answers the request.  And what I will do is I will just, I don't have to read the documentation.  I don't have to do anything.  I can just take this and create a type provider.  And it will be a type provider for a Star Wars person.  And it will be a JSON provider.  And I will give it the URL.  Create this and I have basically everything I need right now.  So now I can write a function to get me information about any person in Star Wars.  So get person.  And I will take an ID as a parameter.  And now I will just call person load.  And I will give it the request that I want.  And I will give it the ID.  I need normal brackets here.  And that's it basically.  Now I can have a look at the first person.  I need to change this into string.  And now I get the information on the first person.  And again, it returns a JSON, which is not very readable.  So I can just type p. and get all the information here in my intelligence, including the types  of all the elements.  So for example, I can look at who the first person actually is.  So I will look at name.  I know that it's a string.  And it's Luke Skywalker.  Do you know what's the, let's say, his mass?  How much he weights?  Did you ever want to know something like this?  Well, he weights 77 kilos.  A lot of very important information.  But what I wanted to show you particularly is that basically in these three lines, I  have everything that was there in the full big wrapper.  And I don't have to care about it at all.  I don't have to read any documentation, write any methods.  This is all I need.  So I actually went on and wrote a full wrapper around the API.  And you can see it's not very imaginative code either.  It's just defining the type providers and defining some of the functions.  And that's it.  That's everything.  And on the same day that I discovered the Star Wars API, I actually just went back and  sent a pull request and now there is an F sharp wrapper.  Thank you.  And in case you were wondering other random things about Star Wars, I can show you some  more information.  For example, do you know what's the most common eye color in Star Wars characters?  You always wanted to know this, right?  So the first most common eye color is brown.  So for example, Yoda has brown eyes.  Who knew?  The second most color is blue.  And the third most common color is yellow.  So Darth Vader has yellow eyes, all the important information.  And you can also see, for example, in episode four, the original Star Wars, Luke Skywalker  once walks into a prison cell and Princess Leia says, aren't you a bit short for a storm  trooper?  So we can see, so is actually Luke Skywalker a bit shorter than average Star Wars character  or not?  Wow.  It's very easy.  I just get his height, which is 172 centimeters.  And I also downloaded information on all the other characters.  And it's 174 centimeters on average.  So yes, Luke Skywalker is shorter than average.  So she was right.  So all the important information that you always wondered about, right?  So well, let's move on.  So if you want to look at it, it's on my GitHub and you can get to it from the Swarovie  website as well.  And play with it.  It's a lot of fun information.  So now what I had right now, when I ran all the code that I was describing, I had characters,  I had the scenes, and I knew that all the characters are actually appearing in the films.  Then Star Wars was bought by Disney and then I came across this analysis and they actually  analyzed 2,000 screenplays and they were looking at how much women characters speak and how  much male characters speak.  And they were just comparing scenes and how much screen times they get.  And they specifically looked at Disney.  And it was sort of depressing because in almost all the Disney movies, men speak much more  than women, etc., even if it's about princesses and things like that.  And because I had almost the exact same data, I thought, well, maybe I can replicate this  using my Star Wars dataset.  And again, type providers.  Because what they did in this analysis is that they went onto IMDB, extracted a list  of actors playing in the individual films, and then they looked at them if they are male  or female.  And I can do exactly the same thing with my type providers.  Because there is not only a type provider for Jason, there is a type provider for HTML  as well.  So IMDB, HTML type provider now.  So let's just open everything again.  And as you can see, I'm also giving it just the URL of some page describing a film.  And this is actually from episode 7, so I will load it now.  And let's...  I don't even have to look at the website.  Now I can do episode 7.  And I know that there are some lists and some tables.  So I will look at the tables.  And the tables are these.  I don't have to go through the website at all.  I see everything here in F-sharp.  And this is all the code I really need to access it.  So I can look at the cast.  And they have the cast in credits order verified as complete.  And here I have a nice printing function to actually look at it properly.  So these are all the people that play something in episode 7.  And for example, did you know that Daniel Craig plays a stormtrooper?  He's somewhere here.  Yeah, Daniel Craig plays a stormtrooper and he's uncredited.  So this gives me quite a lot of information.  And again, it's in my Rappel.  I don't actually have to look at the website and go through it.  So with a bit more code, I put together this graph.  So this is comparing episode 1 to 7 based on what's the percentage of scenes where,  what's the percentage of dialogues with men and what's the percentage of dialogues with  women.  And actually, sorry, I put women and robots together because they are like other genders.  Because otherwise, it would be even worse.  So episode 7 indeed has more women speaking, but still almost 70% of the dialogue is men.  Still better.  The worst one is actually episode 4.  But enough of this.  I had right now, as I said, all the characters and their relationships and where they speak.  So I decided to put together a social network by putting together characters that speak  in the same scene.  If they do, they are connected by a link.  And I can visualize this.  And because, well, what do you do if you want to visualize something nice and put it on  the web?  You use JavaScript.  So I went for D3JS, which is an amazingly powerful library in JavaScript for visualization.  You can write anything there.  The downside is that you can write anything there.  Because it's so powerful that you can't really learn it.  You have to go online and look at various examples of what people did with it before  so that you can copy it and put your own data into it.  Anyway, I don't really want to use JavaScript.  And as I said, I want to do everything in Fsharp here.  So there is this new project called Fable.  So it's using Babel.  This is the current logo.  And that allows you to write code in Fsharp and translate it into JavaScript.  I think it's called transpiling.  Is that the correct term?  So it's a transpiler.  And it's actually really neat.  It's a very new project.  I think it started about six months ago.  And here on the right-hand side, you can see the code in JavaScript from an example on  D3JS that I just downloaded from the web.  And you can see that most of the code on the left-hand side, it's the code in Fsharp.  That's actually calling the D3JS library.  And you can see that it's very similar, actually.  Right?  Well, instead of var, you have let.  But otherwise, all the function calls are very similar.  Sometimes, you have to deal a bit with types, but it's not very painful.  And it allows you to basically just translate your JavaScript code into Fsharp.  And you can call any functions from Fsharp and have all the type safety and everything.  And I just wanted to show you also the translated code.  So this is how it translates the code into JavaScript.  And you can see it, for example, this is calling something called the first layout of networks.  And it's actually very readable.  Like there are no thousands of underscores everywhere.  And you can go there and see what's happening.  And that makes any debugging so much easier.  So I actually went on and did all the things in Fable.  And if you want to play with it, there are many examples online by playing games from  Fsharp, translating it into JavaScript.  There are also some examples with Node.js.  And it's really nice, really.  So let's go to the actual social networks that I promised.  So this is the social network of all the Star Wars movies put together.  And because I promised JavaScript, here it is.  So this is an interactive visualization of Star Wars movies.  Let's make it a bit bigger.  So you can see that the big black note is actually Darth Vader.  And I guess you can already see some patterns emerging.  Whenever you are working with data, try to visualize them because that gives you so much  information.  So I guess right now you can probably guess that on the left-hand side are the new prequel  episodes.  And you can see that the social network is a mess.  You can see that there are so many nodes and it's very dense.  Then in the center are the original episodes.  And on the right-hand side is the episode 7.  We can even look at the episodes individually.  So this is the first episode of the prequels.  And you can see it's quite densely connected.  It has several main characters.  I tried to color them.  So this is, for example, Qui-Gon.  This is Anakin.  This is Jha-Jha.  And when we compare it with the episode 4, the original Star Wars, you can probably immediately  see some difference.  The network is much sparser.  It has quite a few, only a few major characters, and they are connected to each other.  And if you compare it with Force Awakens, the social network is bigger.  But there are not that many characters as in the prequels.  So that already tells us something about the structure of the story and how understandable  it is probably.  And we go even further, we can quantitatively compare the different episodes.  So the first thing that you can think of is how large networks are.  So this is just a graph showing a number of characters.  And these are only the characters that speak at least in two scenes.  And only the characters that are explicitly named.  So if someone is just a Stormtrooper, I didn't include them there.  And yeah, episode 7 has the most characters.  And then the original episodes have fewer, about 20 main characters.  And then in episode 7, it jumps up again up to 27.  So let's hope they don't continue with this trend.  Otherwise, they will get to the episode 1 territory, and you probably don't want to  go there.  And then there are various scientific methods to compare networks.  And the first one is called density, which sounds fancy like a density of a network.  But what it really does, it just tries to compare the number of connections that are  in the network with the number of connections that could be potentially there.  So if you want an equation, this is it, just divide the number of existing connections  in the network by the number of total connections that could be there.  And when we look at this, actually, the episode 1 and 2 have the lowest density.  Because they have a lot of characters that are only vaguely connected, and they have  all these debates in the galactic senate that are not very interesting.  And actually, interestingly, episode 6 has quite a high density, and that's maybe because  I didn't include the Ewoks in there.  And episode 7 actually has about the same density as the original episodes, and so does  episode 3.  And actually, if you look at IMDB, you will see that episode 3 has the highest rating  out of the prequels.  So maybe density has something to do with the quality of the story as well.  And you can look at other measures.  One of them is the clustering coefficient, which tells us how locally connected the network  is and how much the characters actually speak to each other.  So if you look at this green guy, he has three neighbors.  And you can see if the three neighbors are all connected to each other as well or not.  So for this one, there are two of them connected.  For example, for this one, all his friends are connected.  So that tells you how connected your networks are.  And in terms of the story, this basically means if the story is following one character  that just interacts with other people and the other people don't talk to each other,  then that will have a very small clustering coefficient.  And if the story follows a group of people that talk together and interact with each  other a lot, then it will have a large clustering coefficient.  And this is the equation.  We don't actually have to care about it very much.  I don't want to go into details.  But if I plot it for all the different episodes, again, episode one has the lowest clustering  coefficient because there are all these weird characters that are not really talk very much.  And what's nice is that episode seven has about the same clustering coefficient as the  original episodes.  And actually, episode three here has quite a small clustering coefficient as well, which  is interesting.  It tells us something about the story.  And I'm not claiming that small clustering coefficient means a bad story.  I think it makes sense in something.  For example, if the main hero is going through some obstacles and meeting other characters  that are helping him on the way but don't really interact with each other, that can be  a good story.  But I think in Star Wars, it actually tells us something about maybe how the network is  structured.  And actually, I think in this case, it roughly correlates with the quality as well.  And then we can look at local characteristics in the network.  So the first one, the most basic one is degree.  And that's just by looking at how many connections a character has in the network.  So for example, this guy, he's very important because he has six connections in this network,  a small one.  But this guy is less important because he has just three connections.  And that tells us something about centrality of the characters in the social network.  So this just represents how many characters each one of them speaks to.  And yeah, it's just basically number of links that are outgoing or incoming into a note.  And I want to compare it with another measure of centrality in a network, which is called  betweenness.  Because there are many measures of seeing who's the most central in a network.  And the degree just tells us if someone talks to a lot of other characters.  But betweenness tells us how important a note is for communication in the network.  So I will explain again.  So if we look at this guy and these two other guys, the only way they can communicate with  each other is through the green guy because they don't know each other directly.  And these guys, on the other hand, if they want to talk to each other, only one half  of the communication would go through the green guy because they know each other through  some other person as well.  So I can, for example, ask, so if Princess Leia wanted to talk to Jar Jar Binks, who  would she have to go through to pass a message to him?  So that tells me how important a character is within the story.  Because some characters may be important just in one part of the network, but if a character  speaks to a lot of different characters across the whole episode, then that means he's probably  more important to the story.  So again, this is the equation.  And now it's a bit more complicated because for each pair of notes, we look at the number  of shortest paths between them and then look at how many of these shortest paths go through  the specific node.  And we sum it over all different pairs of nodes in the network.  And it's already quite hard to compute, right?  Because you have to compute all the shortest paths between all the nodes in the network  and then look at how many of them pass through the node that you are interested in.  And if you are doing anything with data, there is always a package for it in R. So there  is a library in R called iGraph.  And if you are interested in betweenness, there is a function called betweenness.  And if you are, as I was working in Fsharp, you can do something like this.  Just call iProvider, RProvider.iGraph, and call the betweenness function from R. And  again, you get intelligence and everything.  And this makes anything like this very easy because you do all the heavy preprocessing  in language that you are more comfortable with or that's a bit more safe than R. And  call the algorithms that are already implemented there.  And I actually do this quite a lot because R is a nice programming language, but just  for data science.  If you want to do any general programming, then it can get really, really painful.  And in Fsharp, you get all the safety of type inference.  So it really helped me when I was writing my parser, for example.  And I just knew all the heavy data science in R and I can pass data to it directly from  Fsharp.  So let's look at the centrality.  So who is the most central in episode seven, let's say?  So actually the person who's having the most connections is Spoh.  He's one of the resistance pilots.  And that's because he talks to a lot of people across the whole network.  And he talks to all the other resistance pilots as well.  And there are quite a lot of them.  And the most second central is Finn because he also talks to a lot of other people.  And then Han, Chewbacca, and VV8.  And for VV8, this is just an estimated number because I didn't find him in the screen place  as I was explaining.  So does this tell you something about who's actually important in the story?  Sort of.  But you might be missing some of the main characters in this.  So this is actually a bit weanness.  And now you can see that suddenly Kylo Ren and Ray jumped up a lot.  And that's because they are more important to the actual, they are more important by  connecting different communities within the network.  So for example, Kylo Ren is very important because he is one of the few people that talked  to Han and Chewbacca and Ray, but he also talked to Snoke, which puts him into the center  of the network a bit more.  But also Poe talked to him, which already gave him quite a boost in betweenness as well.  So this tells you a bit more about who's important in the actual network.  And if now you might be asking, so who's the most important across the whole Star Wars,  right?  So before the new episode, this is just Poe celebrating that he's the most central.  So who is the most central overall across the whole Star Wars universe?  Before episode seven, it was this guy because he was talking to quite a lot of people in  the prequels as well as in the original trilogy, a little, although he appeared physically  in only the first one of them.  And also what I did in this analysis when I got this guy, it was because I just looked  at the names in the screenplays.  And actually they are making a very big distinction between Anakin and Darth Vader because they  never appear in the same scene, so they appear like two completely different characters.  And when I added the new episode and also merged Anakin and Darth Vader, still Darth  Vader is ruling the galaxy, he is the most important one.  Everyone else is just lying there dead.  So also I made a quick sample with Neo4j.  So if you want to learn something about Neo4j, I have a link at the end.  You can go there and play with it because it's a very nice data set and you can just  extract connections between people.  And what's important here is that you actually understand what's going on because all the  people have names that you recognize.  And that makes any learning so much easier than if you have just some anonymous customers  or something like that.  You can understand what's happening.  So this is for example just looking at characters that play in the same movie and how they are  connected to each other and computing the degree.  Yeah, degree just within a single movie.  And you can see that Neo4j is actually quite readable because here I'm just looking at  names of characters that appear in...  Yeah, I'm looking at episode four, New Hope, and I'm looking at characters that talk to  other characters that appear also in New Hope and looking at the number of scenes they spoke  together with in.  So this is how you would do it in Neo4j and you can play with it.  And it's not just completely a toy example.  I did something similar by analyzing Twitter and I actually analyzed the social network  around F-Sharp, the F-Sharp Software Foundation, which is the home of open source F-Sharp.  And it has this Twitter handle, F-SharpOrg.  And I looked at all the other Twitter handles that are connected to it.  And then I was looking for the most central people there.  And actually this is the order based on degree.  You can see that the first one is Don Sime, who is actually the author of F-Sharp.  So if you are an alien and come to Twitter and you say, oh, so who is the most important  in C-Sharp, you can do something similar.  I don't know who you get.  But it tells you something about the actual network and some meaning about it.  And the second one, it's, I think it's the official Microsoft side of F-Sharp.  The fifth one is the official side of the community of F-Sharp.  And the fourth one is Thomas, who's speaking tomorrow, if I'm correct, in the morning.  So if you want to see some important people in F-Sharp, Twitter says you should go see  him.  And these are information from November 2014.  So it might have changed.  But the problem with analyzing data from Twitter is that it takes an awful long time to download  because there is all the rate limiting.  So I didn't replicate it afterwards.  But I might do that because, well, it tells you something about how people communicate  with each other and what's happening.  So if there are any changes, I can see it probably.  And also another example, well, this is quite a famous example in network science.  And it was a company in Hungary where they had two factories and some headquarters.  And they were having problems because there are always these rumors spreading in the factories  and they had no idea why and what's happening because the headquarters, they were just issuing  orders and issuing messages.  But there are always rumors and they were not true.  So they actually called, well, in these olden days, they called social scientists who talked  to people and said, so who do you talk to if you want to get more information about anything?  And they named a few people.  And this is the social network that they constructed from it.  And what's interesting is that, well, you can probably see that I think the pink one  is the headquarters.  The other one are the factories.  And the most important note is here.  It's not in the headquarters.  And that was because there was this guy who was doing health and safety.  And he was actually traveling around the factories and talking to everyone.  So he was the most important person in the company for communication.  So now they knew that if they want to spread some kind of information, they have to talk  to him because he will spread the correct information to the others.  And what you would do now, you wouldn't get social scientists to talk to people.  You would just explore the emails that are sent within your company or Slack messages,  like who replies to whom, things like that.  And because you have access to all this information, you can actually do that.  And then you can look at these simple measures, like betweenness and degree centrality, things  like that.  And you immediately get some information that might be actually useful for you.  So as I mentioned, you can do social network analysis.  And it's actually a lot of fun.  And you can look at how you communicate on Slack, who sends emails to whom.  And you can also analyze supply grades.  If there is a blackout in some part of your network, what parts will get affected as well.  So network science is actually very important.  And at the end here, I have biological networks.  And that's actually the area where I work because I look at how genes interact with  each other, what protein binds to what region of DNA.  And then I can see what's happening, how they are interacting with each other.  And this is really not just a toy example, because I was reading a paper the other day  and they were actually looking at the betweenness.  And I was like, ah, I know what betweenness is.  I did it in Star Wars.  And here they were claiming that genes that are important in cancer, that are tumor suppressors  or oncogenes, that they have higher betweenness in the biological networks.  And because I was analyzing the Star Wars networks before, I knew almost exactly what  it means in terms of the actual network, because I got a feeling for it by looking at very  fun data set.  So I want to encourage you to actually play with data, because you get more of it than  just some fun fights about Star Wars.  And right now I went through quite a lot of things.  I went through script parsing in a functional way.  You saw the active patterns.  I was calling R and JavaScript from F-sharp.  I think that's actually the future of data science.  Just call everything from everything, because there are different tools in different languages.  I was showing type providers.  I showed you the HTML type provider and JSON type provider.  And there are many more.  There is one for SQL, there is one for...  I can't even think of all of them.  I use the CSV one, because CSV is the format for data science.  And I would really like to encourage you.  If you are interested in data science, if you have seen some of the talks on R, for  example, then you might be thinking, well, maybe I should go into data science or start  with fun data sets.  Because by analyzing them, you know what's happening there.  And you get insights when you actually see some of these algorithms applied in real world.  And this is, I think, a great way to learn data science.  If you want to know more about F-sharp, I put some links there.  I have the slides online already.  And these are some of the Star Wars resources that I put together.  So, yeah, you can read all the scripts, even if you are not interested in any quantitative  analysis.  You can play with the Star Wars API.  You get all sorts of very important information.  And all the information that I was showing you is actually on my GitHub.  And you can even play with the social networks.  And I saw some people doing actual social network data science, playing with it and trying  to, for example, overlay the different episodes against each other to see who corresponds  to whom in the social network in episode seven, et cetera.  And I have some blog posts about this.  And as I mentioned, I have a Neo4j demo.  And, yeah, play with data.  And this is actually the website where I put the slides.  So go to eval.inghg.com slash Star Wars talk.  And they are all there.  And thank you.  Are there any questions?  So, any questions?  I can't actually see you properly.  I can't see anything.  Well, if there are no questions, then it's time for lunch, I guess.  Thanks, guys.  you
The_Force_Awakens_Evelina_Gabasova_0_59.png The_Force_Awakens_Evelina_Gabasova_60_107.png The_Force_Awakens_Evelina_Gabasova_108_443.png The_Force_Awakens_Evelina_Gabasova_444_1367.png The_Force_Awakens_Evelina_Gabasova_1368_1571.png The_Force_Awakens_Evelina_Gabasova_1572_1619.png The_Force_Awakens_Evelina_Gabasova_1620_1763.png The_Force_Awakens_Evelina_Gabasova_1764_1823.png The_Force_Awakens_Evelina_Gabasova_1824_2015.png The_Force_Awakens_Evelina_Gabasova_2016_2183.png The_Force_Awakens_Evelina_Gabasova_2184_2363.png The_Force_Awakens_Evelina_Gabasova_2364_3131.png The_Force_Awakens_Evelina_Gabasova_3132_5351.png The_Force_Awakens_Evelina_Gabasova_5352_6515.png The_Force_Awakens_Evelina_Gabasova_6516_6575.png The_Force_Awakens_Evelina_Gabasova_6576_10571.png The_Force_Awakens_Evelina_Gabasova_10572_10619.png The_Force_Awakens_Evelina_Gabasova_10620_10667.png The_Force_Awakens_Evelina_Gabasova_10668_10991.png The_Force_Awakens_Evelina_Gabasova_10992_12011.png The_Force_Awakens_Evelina_Gabasova_12012_12071.png The_Force_Awakens_Evelina_Gabasova_12072_12863.png The_Force_Awakens_Evelina_Gabasova_12864_13103.png The_Force_Awakens_Evelina_Gabasova_13104_13235.png The_Force_Awakens_Evelina_Gabasova_13236_13439.png The_Force_Awakens_Evelina_Gabasova_13440_14627.png The_Force_Awakens_Evelina_Gabasova_14628_14951.png The_Force_Awakens_Evelina_Gabasova_14952_15899.png The_Force_Awakens_Evelina_Gabasova_15900_16655.png The_Force_Awakens_Evelina_Gabasova_16656_17015.png The_Force_Awakens_Evelina_Gabasova_17016_17939.png The_Force_Awakens_Evelina_Gabasova_17940_18371.png The_Force_Awakens_Evelina_Gabasova_18372_18695.png The_Force_Awakens_Evelina_Gabasova_18696_18743.png The_Force_Awakens_Evelina_Gabasova_18744_18803.png The_Force_Awakens_Evelina_Gabasova_18804_18851.png The_Force_Awakens_Evelina_Gabasova_18852_18959.png The_Force_Awakens_Evelina_Gabasova_18960_20747.png The_Force_Awakens_Evelina_Gabasova_20748_20807.png The_Force_Awakens_Evelina_Gabasova_20808_20951.png The_Force_Awakens_Evelina_Gabasova_20952_21011.png The_Force_Awakens_Evelina_Gabasova_21012_21107.png The_Force_Awakens_Evelina_Gabasova_21108_25175.png The_Force_Awakens_Evelina_Gabasova_25176_25247.png The_Force_Awakens_Evelina_Gabasova_25248_25511.png The_Force_Awakens_Evelina_Gabasova_25512_27635.png The_Force_Awakens_Evelina_Gabasova_27636_27719.png The_Force_Awakens_Evelina_Gabasova_27720_28463.png The_Force_Awakens_Evelina_Gabasova_28464_32195.png The_Force_Awakens_Evelina_Gabasova_32196_34943.png The_Force_Awakens_Evelina_Gabasova_34944_35687.png The_Force_Awakens_Evelina_Gabasova_35688_37859.png The_Force_Awakens_Evelina_Gabasova_37860_37967.png The_Force_Awakens_Evelina_Gabasova_37968_40559.png The_Force_Awakens_Evelina_Gabasova_40560_40715.png The_Force_Awakens_Evelina_Gabasova_40716_41003.png The_Force_Awakens_Evelina_Gabasova_41004_47939.png The_Force_Awakens_Evelina_Gabasova_47940_56651.png The_Force_Awakens_Evelina_Gabasova_56652_56699.png The_Force_Awakens_Evelina_Gabasova_56700_56747.png The_Force_Awakens_Evelina_Gabasova_56748_56999.png The_Force_Awakens_Evelina_Gabasova_57000_58067.png The_Force_Awakens_Evelina_Gabasova_58068_58403.png The_Force_Awakens_Evelina_Gabasova_58404_68039.png The_Force_Awakens_Evelina_Gabasova_68040_69887.png The_Force_Awakens_Evelina_Gabasova_69888_71159.png The_Force_Awakens_Evelina_Gabasova_71160_71231.png The_Force_Awakens_Evelina_Gabasova_71232_71279.png The_Force_Awakens_Evelina_Gabasova_71280_71327.png The_Force_Awakens_Evelina_Gabasova_71328_72563.png The_Force_Awakens_Evelina_Gabasova_72564_73262.png
Okay, I think it's time to start. So welcome to the Force Awakens with F-Sharp. I don't even know how to pronounce this correctly. So I'm Ewerina Gabasova and I work as a postdoc researcher at Cambridge University in cancer research. So I deal a lot with DNA mutations, things like that, and that's an incredibly complex type of data. And when you look at it, it's really hard to understand what you are doing, and even if you fit some kind of statistical machine learning model, it's really hard to see what's actually happening there. So this is an example of metabolic networks, and you really can't see anything. I can't see anything because I'm not a biologist. So in my free time, I like to play with other data. I suspect they won't be using this title for the actual episode 8. So let's talk about Star Wars. Who likes Star Wars? Yeah, thanks for coming. You are in the right stalk right now. So some time ago, I decided, well, let's analyze Star Wars. I actually started doing this right before the premiere of episode 7, so there was a lot of hype around Star Wars. So I was thinking, so what kind of data can I actually analyze? And well, there are obviously the movies, but it's kind of hard to analyze movies, right? So the next data set that's available about Star Wars are actually the screenplays or scripts. And there are multiple different repositories online where you can go and download scripts of almost any movie you are interested in. And they publish it very quickly. So for example, the script for episode 7 was there, I think, in January. I'm not really sure what's the legal status of that, but if you want to look at it, you can. So I looked at the scripts. And this is an example of a script from episode 4, the original Star Wars. And the great thing about this is that they have a very standard type of format. So you can probably see that there is some title of a scene. And they always start with int or x, meaning interior or exterior. Then there is some description of a scene. And there is a name of people speaking and what they are saying. And because we are programmers, we know that this is actually quite easy to parse. Or if it has this type of structure, we can parse it. So I decided, well, let's look at interactions in Star Wars. And maybe I can even extract social network by looking at who speaks with whom in the scenes. So I went on and downloaded all the script files of all the 7 movies right now. I don't know if I will continue with this in the future, but I have 7 of them now. And I looked at them. And because they are published online, they are usually in HTML. And this is how it looks. And that's also pretty nice, right? There is a pre-formatted HTML block with the scene name in bold. And then there is a description and again, name in bold and in capitals. That's easy to parse. Well, usually. So when I first started doing this, I ended up with a bunch of rejects. Because I wanted to match all the different types of how a name can appear in a script. But because I'm working in F sharp, I would like to show you another way how to parse these things. I know if you have been to Scott Vlaschens talk before me, he was talking about parsers, I'm not doing anything that fancy. But when I looked at the structure of the script, it's actually quite simple. But I, well, this is just an example of the standard format of the script, how it looked. But I want to really write something that I can read. I don't want to deal with how it looks underneath. So what I used something that's called active patterns in F sharp. And there you can write something like this. You just parse a screen. You split the script into elements. And then you can just match each element here with a scene title or a name. And that's very readable. And you don't even have to know what's happening underneath. But I'll show you what's happening underneath. This is something called the active pattern in F sharp. So when you want to match something against some rejects, you don't have to put the rejects into a function that's actually doing the matching. You can just look at, you can define something like this. This is called the banana clips operators. So you can say that a piece of text is either a scene title or a name or something that I'm not interested in. And you can hide all the ugly rejects in here. And by the way, the rejects for names is very complex because in Star Wars, like characters are named anything. You don't get people named C3PO in real life. So this had to be quite complex because it had to match all sorts of dashes, slashes, everything. And I don't want to care about this. For me, rejects are something that it's right only. I don't want to look at it again. So I can hide it in this kind of thing. So what this does is it takes the string as an input and then it matches it against each of the rejects and returns this kind of thing, like scene title and text or the name text or word. And then when I do pattern matching over it, I don't have to care about what it does underneath. And this is readable. When I went back to my original implementation with all the rejects all over the place, I had no idea what was happening there. So for me, this is the way to write parsers in Fsharp or at least for very simple cases. And if you want to play with Fsharp, definitely look at active patterns because they allow you to hide the implementation details. And then you don't have to care about it in the more high level functions. So I thought, now I'm set. It wasn't the case. So I tried to run it on all my seven different script screenplays. And yeah, it was a trap. Because some of the files have completely different structure. For example, the names are not in bold, are not centered. They are at the beginning of a line followed by colon, et cetera. So I had to write another parser for this type of screenplays and it was much more work as usual. And that's a general case in anything data related. You spent 90% of your time just cleaning up the data. And now this also still has some kind of structure because all the names are in capitals at least. You would think so. For example, this one, by the way, the one made on is this scary guy. But if you look at the second letter, it's not an I. It's a lowercase l. I don't know if it came from some OCR system and it just decided that I is a lowercase l. But I, and for things like these, I actually had to put in explicit modifications because I didn't find a way how to deal with this systematically. And that's every time you have to deal with data. So I went through this and I thought, okay, now I'm all set. I have all the characters and all the scenes. It's all nice and easy from now on. Well, no. Because some characters don't even speak in Star Wars screenplays. They are mentioned there but they don't speak at all. For example, R2-D2. If you actually go into the script, they say things like these, like R2 frantically beeps something. Well, thanks. Or Chewbacca doesn't speak at all. So this is Chewbacca barks a comment and Han replies, boy, you said it, Chewy. No, you didn't. So actually, Peter Mayhew who plays Chewbacca, I think he did a very good job playing him because with this type of instruction in the script, what can you do? So I decided, well, when I extracted all the names in all the scenes, I decided, well, I actually have to include these characters because they can't be left out, right? You can't just have Star Wars social network or something without R2-D2 or Chewbacca. So I went on and actually extracted also all the mentions of all the characters in screenplays and tried to estimate how many times would they have spoken if they have spoken there. So I came up with very, very scientific equations. Well, you don't have to care about this, really. It's basically just scaling the number of times a character is mentioned by the times they actually speak. So on average, actually, every character is mentioned about two times as much as they speak explicitly in script. So I just counted how many times, let's say, Chewbacca is mentioned and waited it by how many times Han is mentioned and tried to compute how many times Chewbacca would have spoken in the script. So everything is good now, right? Well, sorry I didn't include some of the characters because there are actually so many characters that don't say a word that Ewoks didn't make it, for example. So the only ones that I included manually in this way were R2D2, Chewbacca, and VV8 from Force Awakens. And then when I look at the characters that my algorithm extracted from all the screenplays, I found out, well, actually, some of the other characters, they appear there but they don't say a word in some of the episodes. So they messed up my analysis again. And if you have seen episode 7, you probably know who I'm talking about. So now I decided, well, I should somehow check if all the characters, because there was such a mess, actually appear in the screenplays. And if there are actual characters, if it's not just some artifact of my algorithm. And I propose a theorem that there is an API for everything. And there is an API for Star Wars. You can go to swapi, StarWarsAPI.co, and you can find an API for Star Wars. So I looked at the website, and it's quite nice, quite well documented. You have all sorts of example requests and what it returns. And you can get information on characters, on starships, on planets, on vehicles, I think as well, everything. And they have a lot of wrapper libraries. So when I looked at them, there was C sharp, Python, R, Java, Go, Ruby. There was no F sharp, but well, maybe I can just use one of the other ones. But then no, no, no. Well, I'm doing this in F sharp. Let's do it properly. And then I looked at, for example, this is the C sharp one. And you can see that the code is not very difficult. Well, these are all just get and set methods and things like that. So there is not actually much going on. And this is just an example of one of the wrappers around, I think, people. And you can see it's 150 lines, and most of them don't actually do anything. So I wanted to show you how we can do something like this in F sharp. And if you have seen any of the F sharp talks, some of them mentioned type providers. So this talk is partly me saying I lost type providers. If you are doing anything with data, type providers are amazing. So I wanted to show you how I can use type providers to write something like all the wrappers that were there in all the different languages in a very small amount of code. So I'll just load the F sharp data library, which contains some of the type providers. And now this is actually just an example request to the Star Wars API. And what I get from this is some document that basically just answers the request. And what I will do is I will just, I don't have to read the documentation. I don't have to do anything. I can just take this and create a type provider. And it will be a type provider for a Star Wars person. And it will be a JSON provider. And I will give it the URL. Create this and I have basically everything I need right now. So now I can write a function to get me information about any person in Star Wars. So get person. And I will take an ID as a parameter. And now I will just call person load. And I will give it the request that I want. And I will give it the ID. I need normal brackets here. And that's it basically. Now I can have a look at the first person. I need to change this into string. And now I get the information on the first person. And again, it returns a JSON, which is not very readable. So I can just type p. and get all the information here in my intelligence, including the types of all the elements. So for example, I can look at who the first person actually is. So I will look at name. I know that it's a string. And it's Luke Skywalker. Do you know what's the, let's say, his mass? How much he weights? Did you ever want to know something like this? Well, he weights 77 kilos. A lot of very important information. But what I wanted to show you particularly is that basically in these three lines, I have everything that was there in the full big wrapper. And I don't have to care about it at all. I don't have to read any documentation, write any methods. This is all I need. So I actually went on and wrote a full wrapper around the API. And you can see it's not very imaginative code either. It's just defining the type providers and defining some of the functions. And that's it. That's everything. And on the same day that I discovered the Star Wars API, I actually just went back and sent a pull request and now there is an F sharp wrapper. Thank you. And in case you were wondering other random things about Star Wars, I can show you some more information. For example, do you know what's the most common eye color in Star Wars characters? You always wanted to know this, right? So the first most common eye color is brown. So for example, Yoda has brown eyes. Who knew? The second most color is blue. And the third most common color is yellow. So Darth Vader has yellow eyes, all the important information. And you can also see, for example, in episode four, the original Star Wars, Luke Skywalker once walks into a prison cell and Princess Leia says, aren't you a bit short for a storm trooper? So we can see, so is actually Luke Skywalker a bit shorter than average Star Wars character or not? Wow. It's very easy. I just get his height, which is 172 centimeters. And I also downloaded information on all the other characters. And it's 174 centimeters on average. So yes, Luke Skywalker is shorter than average. So she was right. So all the important information that you always wondered about, right? So well, let's move on. So if you want to look at it, it's on my GitHub and you can get to it from the Swarovie website as well. And play with it. It's a lot of fun information. So now what I had right now, when I ran all the code that I was describing, I had characters, I had the scenes, and I knew that all the characters are actually appearing in the films. Then Star Wars was bought by Disney and then I came across this analysis and they actually analyzed 2,000 screenplays and they were looking at how much women characters speak and how much male characters speak. And they were just comparing scenes and how much screen times they get. And they specifically looked at Disney. And it was sort of depressing because in almost all the Disney movies, men speak much more than women, etc., even if it's about princesses and things like that. And because I had almost the exact same data, I thought, well, maybe I can replicate this using my Star Wars dataset. And again, type providers. Because what they did in this analysis is that they went onto IMDB, extracted a list of actors playing in the individual films, and then they looked at them if they are male or female. And I can do exactly the same thing with my type providers. Because there is not only a type provider for Jason, there is a type provider for HTML as well. So IMDB, HTML type provider now. So let's just open everything again. And as you can see, I'm also giving it just the URL of some page describing a film. And this is actually from episode 7, so I will load it now. And let's... I don't even have to look at the website. Now I can do episode 7. And I know that there are some lists and some tables. So I will look at the tables. And the tables are these. I don't have to go through the website at all. I see everything here in F-sharp. And this is all the code I really need to access it. So I can look at the cast. And they have the cast in credits order verified as complete. And here I have a nice printing function to actually look at it properly. So these are all the people that play something in episode 7. And for example, did you know that Daniel Craig plays a stormtrooper? He's somewhere here. Yeah, Daniel Craig plays a stormtrooper and he's uncredited. So this gives me quite a lot of information. And again, it's in my Rappel. I don't actually have to look at the website and go through it. So with a bit more code, I put together this graph. So this is comparing episode 1 to 7 based on what's the percentage of scenes where, what's the percentage of dialogues with men and what's the percentage of dialogues with women. And actually, sorry, I put women and robots together because they are like other genders. Because otherwise, it would be even worse. So episode 7 indeed has more women speaking, but still almost 70% of the dialogue is men. Still better. The worst one is actually episode 4. But enough of this. I had right now, as I said, all the characters and their relationships and where they speak. So I decided to put together a social network by putting together characters that speak in the same scene. If they do, they are connected by a link. And I can visualize this. And because, well, what do you do if you want to visualize something nice and put it on the web? You use JavaScript. So I went for D3JS, which is an amazingly powerful library in JavaScript for visualization. You can write anything there. The downside is that you can write anything there. Because it's so powerful that you can't really learn it. You have to go online and look at various examples of what people did with it before so that you can copy it and put your own data into it. Anyway, I don't really want to use JavaScript. And as I said, I want to do everything in Fsharp here. So there is this new project called Fable. So it's using Babel. This is the current logo. And that allows you to write code in Fsharp and translate it into JavaScript. I think it's called transpiling. Is that the correct term? So it's a transpiler. And it's actually really neat. It's a very new project. I think it started about six months ago. And here on the right-hand side, you can see the code in JavaScript from an example on D3JS that I just downloaded from the web. And you can see that most of the code on the left-hand side, it's the code in Fsharp. That's actually calling the D3JS library. And you can see that it's very similar, actually. Right? Well, instead of var, you have let. But otherwise, all the function calls are very similar. Sometimes, you have to deal a bit with types, but it's not very painful. And it allows you to basically just translate your JavaScript code into Fsharp. And you can call any functions from Fsharp and have all the type safety and everything. And I just wanted to show you also the translated code. So this is how it translates the code into JavaScript. And you can see it, for example, this is calling something called the first layout of networks. And it's actually very readable. Like there are no thousands of underscores everywhere. And you can go there and see what's happening. And that makes any debugging so much easier. So I actually went on and did all the things in Fable. And if you want to play with it, there are many examples online by playing games from Fsharp, translating it into JavaScript. There are also some examples with Node.js. And it's really nice, really. So let's go to the actual social networks that I promised. So this is the social network of all the Star Wars movies put together. And because I promised JavaScript, here it is. So this is an interactive visualization of Star Wars movies. Let's make it a bit bigger. So you can see that the big black note is actually Darth Vader. And I guess you can already see some patterns emerging. Whenever you are working with data, try to visualize them because that gives you so much information. So I guess right now you can probably guess that on the left-hand side are the new prequel episodes. And you can see that the social network is a mess. You can see that there are so many nodes and it's very dense. Then in the center are the original episodes. And on the right-hand side is the episode 7. We can even look at the episodes individually. So this is the first episode of the prequels. And you can see it's quite densely connected. It has several main characters. I tried to color them. So this is, for example, Qui-Gon. This is Anakin. This is Jha-Jha. And when we compare it with the episode 4, the original Star Wars, you can probably immediately see some difference. The network is much sparser. It has quite a few, only a few major characters, and they are connected to each other. And if you compare it with Force Awakens, the social network is bigger. But there are not that many characters as in the prequels. So that already tells us something about the structure of the story and how understandable it is probably. And we go even further, we can quantitatively compare the different episodes. So the first thing that you can think of is how large networks are. So this is just a graph showing a number of characters. And these are only the characters that speak at least in two scenes. And only the characters that are explicitly named. So if someone is just a Stormtrooper, I didn't include them there. And yeah, episode 7 has the most characters. And then the original episodes have fewer, about 20 main characters. And then in episode 7, it jumps up again up to 27. So let's hope they don't continue with this trend. Otherwise, they will get to the episode 1 territory, and you probably don't want to go there. And then there are various scientific methods to compare networks. And the first one is called density, which sounds fancy like a density of a network. But what it really does, it just tries to compare the number of connections that are in the network with the number of connections that could be potentially there. So if you want an equation, this is it, just divide the number of existing connections in the network by the number of total connections that could be there. And when we look at this, actually, the episode 1 and 2 have the lowest density. Because they have a lot of characters that are only vaguely connected, and they have all these debates in the galactic senate that are not very interesting. And actually, interestingly, episode 6 has quite a high density, and that's maybe because I didn't include the Ewoks in there. And episode 7 actually has about the same density as the original episodes, and so does episode 3. And actually, if you look at IMDB, you will see that episode 3 has the highest rating out of the prequels. So maybe density has something to do with the quality of the story as well. And you can look at other measures. One of them is the clustering coefficient, which tells us how locally connected the network is and how much the characters actually speak to each other. So if you look at this green guy, he has three neighbors. And you can see if the three neighbors are all connected to each other as well or not. So for this one, there are two of them connected. For example, for this one, all his friends are connected. So that tells you how connected your networks are. And in terms of the story, this basically means if the story is following one character that just interacts with other people and the other people don't talk to each other, then that will have a very small clustering coefficient. And if the story follows a group of people that talk together and interact with each other a lot, then it will have a large clustering coefficient. And this is the equation. We don't actually have to care about it very much. I don't want to go into details. But if I plot it for all the different episodes, again, episode one has the lowest clustering coefficient because there are all these weird characters that are not really talk very much. And what's nice is that episode seven has about the same clustering coefficient as the original episodes. And actually, episode three here has quite a small clustering coefficient as well, which is interesting. It tells us something about the story. And I'm not claiming that small clustering coefficient means a bad story. I think it makes sense in something. For example, if the main hero is going through some obstacles and meeting other characters that are helping him on the way but don't really interact with each other, that can be a good story. But I think in Star Wars, it actually tells us something about maybe how the network is structured. And actually, I think in this case, it roughly correlates with the quality as well. And then we can look at local characteristics in the network. So the first one, the most basic one is degree. And that's just by looking at how many connections a character has in the network. So for example, this guy, he's very important because he has six connections in this network, a small one. But this guy is less important because he has just three connections. And that tells us something about centrality of the characters in the social network. So this just represents how many characters each one of them speaks to. And yeah, it's just basically number of links that are outgoing or incoming into a note. And I want to compare it with another measure of centrality in a network, which is called betweenness. Because there are many measures of seeing who's the most central in a network. And the degree just tells us if someone talks to a lot of other characters. But betweenness tells us how important a note is for communication in the network. So I will explain again. So if we look at this guy and these two other guys, the only way they can communicate with each other is through the green guy because they don't know each other directly. And these guys, on the other hand, if they want to talk to each other, only one half of the communication would go through the green guy because they know each other through some other person as well. So I can, for example, ask, so if Princess Leia wanted to talk to Jar Jar Binks, who would she have to go through to pass a message to him? So that tells me how important a character is within the story. Because some characters may be important just in one part of the network, but if a character speaks to a lot of different characters across the whole episode, then that means he's probably more important to the story. So again, this is the equation. And now it's a bit more complicated because for each pair of notes, we look at the number of shortest paths between them and then look at how many of these shortest paths go through the specific node. And we sum it over all different pairs of nodes in the network. And it's already quite hard to compute, right? Because you have to compute all the shortest paths between all the nodes in the network and then look at how many of them pass through the node that you are interested in. And if you are doing anything with data, there is always a package for it in R. So there is a library in R called iGraph. And if you are interested in betweenness, there is a function called betweenness. And if you are, as I was working in Fsharp, you can do something like this. Just call iProvider, RProvider.iGraph, and call the betweenness function from R. And again, you get intelligence and everything. And this makes anything like this very easy because you do all the heavy preprocessing in language that you are more comfortable with or that's a bit more safe than R. And call the algorithms that are already implemented there. And I actually do this quite a lot because R is a nice programming language, but just for data science. If you want to do any general programming, then it can get really, really painful. And in Fsharp, you get all the safety of type inference. So it really helped me when I was writing my parser, for example. And I just knew all the heavy data science in R and I can pass data to it directly from Fsharp. So let's look at the centrality. So who is the most central in episode seven, let's say? So actually the person who's having the most connections is Spoh. He's one of the resistance pilots. And that's because he talks to a lot of people across the whole network. And he talks to all the other resistance pilots as well. And there are quite a lot of them. And the most second central is Finn because he also talks to a lot of other people. And then Han, Chewbacca, and VV8. And for VV8, this is just an estimated number because I didn't find him in the screen place as I was explaining. So does this tell you something about who's actually important in the story? Sort of. But you might be missing some of the main characters in this. So this is actually a bit weanness. And now you can see that suddenly Kylo Ren and Ray jumped up a lot. And that's because they are more important to the actual, they are more important by connecting different communities within the network. So for example, Kylo Ren is very important because he is one of the few people that talked to Han and Chewbacca and Ray, but he also talked to Snoke, which puts him into the center of the network a bit more. But also Poe talked to him, which already gave him quite a boost in betweenness as well. So this tells you a bit more about who's important in the actual network. And if now you might be asking, so who's the most important across the whole Star Wars, right? So before the new episode, this is just Poe celebrating that he's the most central. So who is the most central overall across the whole Star Wars universe? Before episode seven, it was this guy because he was talking to quite a lot of people in the prequels as well as in the original trilogy, a little, although he appeared physically in only the first one of them. And also what I did in this analysis when I got this guy, it was because I just looked at the names in the screenplays. And actually they are making a very big distinction between Anakin and Darth Vader because they never appear in the same scene, so they appear like two completely different characters. And when I added the new episode and also merged Anakin and Darth Vader, still Darth Vader is ruling the galaxy, he is the most important one. Everyone else is just lying there dead. So also I made a quick sample with Neo4j. So if you want to learn something about Neo4j, I have a link at the end. You can go there and play with it because it's a very nice data set and you can just extract connections between people. And what's important here is that you actually understand what's going on because all the people have names that you recognize. And that makes any learning so much easier than if you have just some anonymous customers or something like that. You can understand what's happening. So this is for example just looking at characters that play in the same movie and how they are connected to each other and computing the degree. Yeah, degree just within a single movie. And you can see that Neo4j is actually quite readable because here I'm just looking at names of characters that appear in... Yeah, I'm looking at episode four, New Hope, and I'm looking at characters that talk to other characters that appear also in New Hope and looking at the number of scenes they spoke together with in. So this is how you would do it in Neo4j and you can play with it. And it's not just completely a toy example. I did something similar by analyzing Twitter and I actually analyzed the social network around F-Sharp, the F-Sharp Software Foundation, which is the home of open source F-Sharp. And it has this Twitter handle, F-SharpOrg. And I looked at all the other Twitter handles that are connected to it. And then I was looking for the most central people there. And actually this is the order based on degree. You can see that the first one is Don Sime, who is actually the author of F-Sharp. So if you are an alien and come to Twitter and you say, oh, so who is the most important in C-Sharp, you can do something similar. I don't know who you get. But it tells you something about the actual network and some meaning about it. And the second one, it's, I think it's the official Microsoft side of F-Sharp. The fifth one is the official side of the community of F-Sharp. And the fourth one is Thomas, who's speaking tomorrow, if I'm correct, in the morning. So if you want to see some important people in F-Sharp, Twitter says you should go see him. And these are information from November 2014. So it might have changed. But the problem with analyzing data from Twitter is that it takes an awful long time to download because there is all the rate limiting. So I didn't replicate it afterwards. But I might do that because, well, it tells you something about how people communicate with each other and what's happening. So if there are any changes, I can see it probably. And also another example, well, this is quite a famous example in network science. And it was a company in Hungary where they had two factories and some headquarters. And they were having problems because there are always these rumors spreading in the factories and they had no idea why and what's happening because the headquarters, they were just issuing orders and issuing messages. But there are always rumors and they were not true. So they actually called, well, in these olden days, they called social scientists who talked to people and said, so who do you talk to if you want to get more information about anything? And they named a few people. And this is the social network that they constructed from it. And what's interesting is that, well, you can probably see that I think the pink one is the headquarters. The other one are the factories. And the most important note is here. It's not in the headquarters. And that was because there was this guy who was doing health and safety. And he was actually traveling around the factories and talking to everyone. So he was the most important person in the company for communication. So now they knew that if they want to spread some kind of information, they have to talk to him because he will spread the correct information to the others. And what you would do now, you wouldn't get social scientists to talk to people. You would just explore the emails that are sent within your company or Slack messages, like who replies to whom, things like that. And because you have access to all this information, you can actually do that. And then you can look at these simple measures, like betweenness and degree centrality, things like that. And you immediately get some information that might be actually useful for you. So as I mentioned, you can do social network analysis. And it's actually a lot of fun. And you can look at how you communicate on Slack, who sends emails to whom. And you can also analyze supply grades. If there is a blackout in some part of your network, what parts will get affected as well. So network science is actually very important. And at the end here, I have biological networks. And that's actually the area where I work because I look at how genes interact with each other, what protein binds to what region of DNA. And then I can see what's happening, how they are interacting with each other. And this is really not just a toy example, because I was reading a paper the other day and they were actually looking at the betweenness. And I was like, ah, I know what betweenness is. I did it in Star Wars. And here they were claiming that genes that are important in cancer, that are tumor suppressors or oncogenes, that they have higher betweenness in the biological networks. And because I was analyzing the Star Wars networks before, I knew almost exactly what it means in terms of the actual network, because I got a feeling for it by looking at very fun data set. So I want to encourage you to actually play with data, because you get more of it than just some fun fights about Star Wars. And right now I went through quite a lot of things. I went through script parsing in a functional way. You saw the active patterns. I was calling R and JavaScript from F-sharp. I think that's actually the future of data science. Just call everything from everything, because there are different tools in different languages. I was showing type providers. I showed you the HTML type provider and JSON type provider. And there are many more. There is one for SQL, there is one for... I can't even think of all of them. I use the CSV one, because CSV is the format for data science. And I would really like to encourage you. If you are interested in data science, if you have seen some of the talks on R, for example, then you might be thinking, well, maybe I should go into data science or start with fun data sets. Because by analyzing them, you know what's happening there. And you get insights when you actually see some of these algorithms applied in real world. And this is, I think, a great way to learn data science. If you want to know more about F-sharp, I put some links there. I have the slides online already. And these are some of the Star Wars resources that I put together. So, yeah, you can read all the scripts, even if you are not interested in any quantitative analysis. You can play with the Star Wars API. You get all sorts of very important information. And all the information that I was showing you is actually on my GitHub. And you can even play with the social networks. And I saw some people doing actual social network data science, playing with it and trying to, for example, overlay the different episodes against each other to see who corresponds to whom in the social network in episode seven, et cetera. And I have some blog posts about this. And as I mentioned, I have a Neo4j demo. And, yeah, play with data. And this is actually the website where I put the slides. So go to eval.inghg.com slash Star Wars talk. And they are all there. And thank you. Are there any questions? So, any questions? I can't actually see you properly. I can't see anything. Well, if there are no questions, then it's time for lunch, I guess. Thanks, guys. you
you how I can use type providers to write and put your own data into it. Anyway, I don't similar, actually. Right? Well, instead of var, the Star Wars movies put together. And because I first episode of the prequels. And you can see language, but just for data science. If you want science in R and I can pass data to it directly so who's the most important across the whole Star constructed from it. And what's interesting is And you get insights when you actually see some this is, I think, a great way to learn data