Making Magic with F# 3.0 Type Providers
Syme, Don
Do you want programming languages which scale to integrate internet-scale information sources like data markets, entity graphs, the Google Knowledge Graph and the information spaces of the modern enterprise? The modern web and enterprise is highly information rich, but our programming languages are information sparse, especially our strongly typed ones. "Information Rich Programming" (IRP) is an emerging direction for strongly-typed language design and implementation, and the innovative F# language from Microsoft is leading the way in this area. In this talk, we'll give an overview of the challenges of strongly-typed IRP against web data markets, web ontologies, databases, services and enterprise data schema. Are information spaces "just" libraries? Can we gives types to "everything", and if so, should we? What does it mean for future languages? We'll demonstrate what F# 3.0 specifically offers in the area of IRP, but also look at how information-richness makes us reconsider programming language and tooling design more generally, and take a look at the themes that recur in this kind of work.
 Hi everyone. Great to be back in Oslo. It's, I came here for a Java conference last time and it's NDC this time. So my name is Don Sein. I'm a principal researcher at Microsoft Research in Cambridge. I, my mate, I guess some people know me as having worked on C-Sharp and the design of the F-Sharp language. I, F-Sharp, I, as one of the things I'll be bringing out about the F-Sharp programming language today is that one of the exciting things that's going on is it's a strong open source community and I see myself as a contributor to that community rather than as a sort of, sort of leader of that community. And just one of the, one of the many people contributing. And so I prefer to introduce myself these days as a F-Sharp contributor who happens to also be a principal researcher at Microsoft Research. So today I'm going to be talking about what for me has been perhaps the most exciting, the most inspiring, the most intriguing programming language feature I've had the pleasure to work on during my time at Microsoft Research. And it's all about the fundamental problem of how we bring data, information, services and programming together and do this at a scale that's suitable for the modern era. And you can use it today in product form in Visual Studio 2012 in F-Sharp 3.0. You can use it through the open source editions of F-Sharp on all sorts of platforms to access all sorts of information services in many, many different kinds of programming settings. But beyond that, it's actual existence in F-Sharp today. I hope it also serves as an inspiration for what programming can be in the future for a whole range of different languages. We've seen this work have, I know through personal contacts and emails, having direct and strong influence on the designers of C-Sharp and the designers of OCaml and of Scala. And I believe through that and through, as it flows through the industry, we'll see this gradually roll out in many different forms and perhaps many of which we might forget the some of the background that comes from here. But that's sort of the role of research is to come up with ideas which spread of their own accord through many different settings in the industry. So I'm excited from both from a research perspective and also extremely happy with what we've made available to you as developers in F-Sharp 3.0 both in Visual Studio and in the open source editions of F-Sharp. So the fundamental problem as we all know is the world is information rich. It's actually a very, very interesting week to give this line about information richness. And it's, I mean, we know the many, many positive sides of information richness and we know some of the negatives that that might bring and some of the challenges that brings to our society. The way I like to put it is, I mean, as we've known for a long time, the web changes absolutely everything. The web changes the whole way we work as individuals. It enables all sorts of experiences that with using devices that we never thought would be possible. And it also, it changes politics, changes communications, changes many, many things. But this talk is really about how it can go down to the very base level and change some assumptions we make about programming languages and about how we think about the role of programming languages in the world, especially strongly typed programming languages. And the context is the information revolution. This is just one of many ways of visualizing this. This is taken from programmableweb.com about the rise of open APIs on the web. You can see the numbers below here. There's basically an exponential rise in the sort of digital API context that's available on the open web. And we don't expect this kind of thing to stop anytime soon. The size, if you like, the sort of, you know, the magnitude of the information context around us will just continue to grow and grow. And this is the world we have to interact with as programmers. So modern programming is intensely information rich. You see manifestations of that in data science. You see it in programming for devices in that these are now connected always, nearly always connected through to some kind of set of information sources. You see it in the enterprise for a long time. The enterprise has sort of been a microcosm of what is now happening sort of universally across programming. But our languages are information sparse. They don't in themselves contain any information. They don't actually contain for our strongly typed languages. They don't contain representations of information coming from the outside. You won't in a C-sharp library space, you won't find a definition of a chemical element or an amino acid or a football player or any or genome or any of the information sources that people actually have to deal with every day. So instead, our languages contain these very generic information adapters like XML or text or bytes or JSON sort of APIs to help us do sort of untyped generic manipulation of data. And that's great for many purposes. But for many purposes, we need to program against precise data structures. And those need to come from somewhere and be brought into the language from somewhere. So the developer's perspective is in some sense the languages do not integrate information well. If they do, it's sort of either non-intuitive or it's not simple or disorganized or static or high friction. We have to use these generic adapters like web requests or XML readers or JSON. And this is a problem. It's a problem especially for strongly typed languages. Strongly typed languages are incredibly efficient information manipulation devices. Once you have data under the type system of a modern strongly typed language, especially a strongly typed functional programming language, or one with the sort of expressive power of say the.NET and C-Sharp type system, then that's great. Everything kind of goes pretty well. There's still problems. Nulls, for instance, can be a problem in some type systems. But once you have it under the type system, things are good. But if the cost of getting information under the discipline of the type system is too high, then any efficiency you gain by getting information under types is just lost around the edges. And this is one of the main reasons for the big resurgence of interest in dynamically typed programming languages is because you need to interoperate with the outside world more than you need to process things internally in the language. So we need to bring information into the language, integrated into the language at internet scale, strongly tooled and strongly typed. And that's what we're trying to do here. But back in the good old days, the world was neatly separated into statically typed and dynamically typed languages. And everything was simple and the textbooks all were nice and clear. And this has sort of changed a lot because of this problem of integrating with the external world and also the rise of a lot of different kinds of programming as well. So we now see a lot more sort of overlap. And I guess there's a search on in the middle to make statically typed languages more dynamic. You can see that same C sharp 4.0 to make dynamically typed languages more static through, for instance, lots of type inference in JavaScript implementations or to put sort of static facades over dynamically typed languages like in TypeScript. And then there's sort of moderate static typing, some of the guarantees that you get from static typing in limited ways, but expand the use of static typing. And that's what we're doing in this talk here. But before we get into that, I need to tell you about the context that we're doing it in, which is F sharp. And let me, so I'll just run through a little bit of material about F sharp. I'll do this fairly quickly, but just to give you a locator for where we are with F sharp. So these days, I like to separate between two things. One is F sharp as a language and one, the other is a visual F sharp tool. So the visual F sharp tools are part of Visual Studio. F sharp as a language is still designed primarily at, through, at Microsoft and the compiler implementation is done at Microsoft. But F sharp as a language is open source and is cross platform. And this has been a really, I mean, very exciting developments in the F sharp community through the development, through the community of this F sharp software foundation. And this is a software foundation in many ways set up to be orthogonal, to give the reality to F sharp being open source. And F sharp, they see it as their mission to sort of complement what's done by the core language and compiler team at Microsoft and to bring F sharp onto Mac, Linux, Android, iPhone, HTML5 and, and platforms like FreeBSD and even GPGPU programming. And to give sort of a non, I think a non-Microsoft perspective on, on F sharp. I'll talk a bit more about that in a bit. But just to get across that, I think it's worth separating out those two ideas. Visual F sharp tools are part of Visual Studio. F sharp as a language is open source and cross platform. So our mission statement sort of F sharp is strongly typed data immersive functional first language for simple code. We used to say simple code for complex problems. Okay. So actually it's just about simple code when you really kind of come down to it. I mean, you happen to be able to solve a range of complex problems with it. I'll be talking, I'll be giving another talk here today at, just after lunch, talking about functional first programming more broadly and how, how you can succeed with functional first programming in, in industry. And I'll dig into this a bit more of what sort of roles it can play. But code simplicity is absolutely at the core of a lot of what we do in strongly typed functional languages and in F sharp in particular. So there's a function to swap two elements of a tuple and in C sharp that's okay. You can write this in C sharp, but it just kind of gets a bit longer and you take the rotations of a tuple XYZ and you get three, three rotations and yeah, it's fine. But you know, what do you do in C sharp? It's like, it's actually annoyingly hard to write out this kind of simple data manipulation lists and tuples. And it's just one example of many, many examples of F sharp coding which drop out as sort of more simple, more declarative, better properties in terms of robustness and particularly because of the lack of nulls in, in, in F sharp coding. And there's another one reducing FXYZ. You can do the same sort of thing in C sharp. But again, it's just a bit, bit longer and not as declarative. Simple code defining a type, a class in F sharp of one line, it might go on for many lines in C sharp. And you can see that in our data access work which will enable us later on in this talk to write things like this. Get me the, get me the population age of zero to 14 is a percentage of total of Brazil. You know, it kind of doesn't get more simple than that. We're integrating this information from the World Bank Information Service. And it's just about this, it's sort of bringing out that what we're doing in this talk is following on with the general theme. The general mission statement of F sharp about being about simple code carries through to simplicity in data access as well. One thing I'll, I, I, I like to, to, to bring up at this point is why you might not want to use a functional programming language. And this is actually a very humorous piece of satire about, about why you, why you might not want to use a strongly type functional language. Like if you get paid by the line, for example, that's a, you know, then you, you might prefer to write this sort of code. I compare it, you know, with a couple of lines of F sharp code and so on. Love me some curly braces. I can't do without them. Okay. So if you're that kind of program, if you really just can't do without your curly braces, then there, you know, it might not be the right setting for you, but you know, you probably, I reckon you probably can live without those, those curly braces and you can work on that. Okay. How does F sharp get used in practice? I'll, I'll be talking more about this in my other talks. So just a couple of quick examples. One is, for example, application to balance the national power generation schedule for a power company in the UK. It's very common that F sharp gets used as a, as a component inside a larger dot net or maybe a dot net service with a JavaScript HTML5 front end or just within a larger dot net application. One of the interesting things about F sharp is that, is that property that because it's so easy to interrupt not only from F sharp to the rest, for F sharp to access dot net libraries and data services as we'll see in this talk, but you can interoperate from dot net to F sharp very easily, which means you can encapsulate your use of F sharp into however many components you happen to be sort of right for you to use. And that tends to start off as one component and then people expand out their use of F sharp. So Kaggle, for example, Kaggle is a data science company. They run competitions for a sort of crowdsourcing of data analysis. You can enter these competitions and try and do the better data and statistical analysis than other people over these data sets. Kaggle themselves who run these competitions use F sharp for their core data analysis algorithms. And then they, as I said, you start off with some core data analysis algorithms and then you move more and more of your application into F sharp. And the usual things that people report, and this has been a long standing thing with strongly type functional programming, is fewer bugs, easy to refactor, easy to read, and stronger, and because the strong typing gives the better robustness and fewer bugs. Okay. So as I said, F sharp is changing a bit. There was this view, I think, before the F sharp was for Windows. That's just not the case anymore. F sharp runs on many platforms. There's also, I think, a view of Microsoft make F sharp, and that's actually changed a lot. F sharp has many contributors now. The core compiler language is still done by Microsoft, the team at Microsoft, but for Windows and that gets taken and the other contributors take it to other platforms. I think there was a, when we came out with F sharp, when Microsoft come out with something, we have a bit of an overly strong, a bit heavy, I think. So we came out with F sharp from Microsoft's perspective, and that's actually changing to be many different perspectives on what the programming language is for through F sharp.org. And yeah, I guess it's sort of a, it's instead of seeing F sharp as being something that kind of comes from Microsoft for Microsoft's purposes, there's now many different contributors, some of which are Microsoft, some of which are Microsoft research, and in fact Microsoft research have somewhat different goals than Microsoft in contributing to F sharp. Our role is to partly to take programming forward through a language like F sharp. We have everything flowing in from Nougat into the F sharp environment. You can access all the Nougat packages, 11,000 of them from F sharp. There's lots of good community sites. We have Xamarin is a company which provides F sharp support on the iOS and Android platforms along with them primarily C sharp support, but an F sharp option as well, and quite a few startups contributing to F sharp as well. Just to give an example of the exo system around F sharp, that is, not these aren't intrinsically F sharp packages, these are.NET packages, and one of the things about F sharp is it kind of lives in this exo system, is embedded in this exo system of.NET. If you want to access Amazon web service,.NET SDKs, you can do that, and you can access all the great services from Amazon. If you want to access the Azure.NET SDKs, you can use those from F sharp, and they're all open source as well, and you can go and do your cloud computing with either of these platforms. You want to learn more about particular areas of F sharp, data science, machine learning, cloud or data access and so on. Go to F sharp.org, there's lots of good information. You can contribute to those pages. You just send a pull request on the GitHub source for these pages, and then the community organization will integrate your information into that setting as well, so you can get involved with the community foundation. Okay, so back to the main topic, bringing information into the language, at internet scale, strongly tooled and strongly typed. Okay. Right, so let's start with a small piece of information, which is freebase.com. So freebase is an entity graph. You can more or less assume this is taking all of Wikipedia and doing entity extraction, looking at those structured panels of information that are down the right, for instance, that are given organized view over, say, all the different individual entities, individual pages in Wikipedia and actually elsewhere as well, and integrating those into a single web database slash ontology slash service, whatever you want to call it. It's a pre-base, has been acquired by Google and it's a predecessor to what's called the Google Knowledge Graph, which is sort of an even bigger version of freebase. Freebase has got a publicly accessible API, which is what we'll be using. The Google Knowledge Graph, as far as I know, doesn't have that, but it is much bigger. But it's essentially, again, everything that Google sees on the open web run through a set of entity extraction algorithms and graph database wizardry and a whole lot of other amazing stuff that I don't know much about. The end result is an extraordinary sort of source of organized, schematized information about the real world. For instance, you want three million books that you can query and search, then this is a good place to find them. If you want whole databases of TV and film and music and sports and government and just everything, you can find thousands and millions of topics under here. A lot of people, a lot of different companies are getting into entity extraction. This information is useful for all sorts of things. One of the things it's mainly used for is search to provide those lovely little structured panels you get either from Bing or from Google when you type in something into the search engines. They kind of use it to give a more entity-based structured view of the world around us. But wouldn't it be nice if we could just access all of this information from the programming perspective and do it in a sort of strongly typed and organized way? So we might, for example, be trying to do something like, where's my talk gone? Okay, now I've got to skip through. We might be after something like accessing the chemical elements as part of a chemistry app for a phone or for a web or for an educational application. And we'd like to access some basic information about chemistry. Okay, before I begin, I'll just do a little bit of F-sharp programming here. What I'm using here is F-sharp interactive in Visual Studio. I'll just put this down the bottom here. I've created myself an F-sharp script and I've loaded up some scripts up above. The most important one of which here is this one which is called show, which provides me with a function called show, which allows me to take some data. I've created here the numbers from 0 to 99 and I can call the show function on it and display the data over in the right-hand window. I can also do the same thing by piping, using this piping syntax, if I change that to 59 and then show the numbers over on the right. Now, I'm taking the numbers and I'm piping them into the show function here. You'll see me use that quite a lot. I can also use this show grid function to show a more sort of tabular display. And for instance, if I get the table of squares and I do a show grid over on the right, you'll see the values coming out. For some reason, it's showing duplicates when I'm not sure why that is. That's a bug in the show function, in the show grid. I'll also do some charting, so we'll take that data of the table of squares. So this is the numbers for i from 0 to 99 generating i and the square of i. And that's our data there, that's our data set. And we can pass that data set to the charting library, which will then just display the data set in the obvious way. Get rid of this. So let's take a random point cloud, for example, so 10,000 points where we're multiplying two random numbers here, across here, and you see them drift down to an average of about a quarter down here. Okay, just to show how we might do interactive data work with F-sharp, in this one we're going to take a stream of events coming from this window over here. And as the stream comes in, we're going to collect up a data set and display that data set in the chart over on the left, so that as I move my mouse over on the right, we're accumulating a data set from the events on the outside and displaying the data set over on the left. Okay, so that was to take the mouse move event to extract out the x and the y coordinates from the mouse move and to create an updating chart from updating chart.line. This is using, if you're wondering, this is using the F-sharp dot charting library, which is, and in the latest edition of F-sharp charting, you'll see that updating chart has actually been renamed to live chart if you're going to try and do this at home. Okay, so to take another one here, we're going to take, again, showing how functional programming plays with working with data and information. We're going to take the mouse move events from over here. We're going to extract the y coordinates. We're going to sample them every 30 milliseconds. We're going to take a 3,000 millisecond, three-second interview, interval over that, and collect up that data set, updating data set, and we'll have that displayed over here, and you can see our data set has been generated. And I'd like you to think, how long would it take you to do that kind of data specification and transformation and visualization in an object-oriented kind of way? Okay. So this is fundamentally, you want combinators like sampling, you want them like taking a window, that's exactly what you think about when specifying the problem is, I want to extract out the y coordinates, I want to sample, I want a window time interval, and I want an updating chart of that data set. Right, but let's look at information integration. Okay, so what we've done with F-sharp 3.0 is really, part of the magic of it is that it's just such a simple change to the language, okay, which is that instead of referencing a library, we also use to the idea of referencing a DLL, okay, we know that in.NET, or.NET DLL contains a set of types, a set of classes, a set of namespaces in classes, and we know that in the Java context, you reference a JAR file, it's the same thing, it contains some types, okay. So why not move to a provider model for libraries instead of a fixed set of types? So instead of referencing just a finite collection of types, we're actually referencing a component which computes types on, types and the contents of types on demand, okay. So we'll go through what that means in more detail in a bit, but effectively it's just doing what we often do in computing, which is when we have something, we turn to use a provider model for that something instead. So when I'm referencing Freebase here, that Freebase.dll only contains one or two types, okay, but those are special components which can compute types on demand, okay, and compute them at compile time and compute them in the development environment. This DLL acts as a plug-in to Visual Studio and it also acts as a plug-in to my execution environment down below. So we're moving to a provider model for types and that DLL here lets us have this Freebase data provider and we give it some static parameters. We tell it how we're going to access the service, so give it the API key to our service, which is known at compile time to access the metadata for Freebase and we give it another parameter which I'll explain more in a bit. And once we have that executing the lines through to this data line, then the type of this data object is now represents the entire universe of information that's available through Freebase, okay. So that's effectively handled to a structured type view of everything on Wikipedia. So that means when we start to navigate the properties of this data object, we're actually sort of navigating not at runtime where we were actually accessing the data, but at compile time where navigating the metadata associated with this service. Okay, so in many cases I haven't actually previously accessed these, I've never looked at that engineering domain before, okay. So you're interested in knowing things about engineering and you're knowing, for instance, what different kinds of batteries are available in the world, then we can at compile time get strong typing and strong tooling to access all the different parts of this information service. So for instance astronomy, we can come through, we can come through to astronomers or to stars here. One moment. Okay, to stars or star systems and all the different tables of information available in the information service about astronomy. So let's take a look at some of the actual data, for instance, we can come through to biology and we can look at the amino acids. The double ticks used in F-sharp are just because the name contains a space which is very common in this kind of working. So let's get the table of amino acids and we have it available over on the right here. Okay, so now this is strongly typed data, although it might look like it is a sort of a dynamic language feature, so if I actually iterate the amino acids here and I look and I try and do completion, we haven't actually iterated these, I'm trying to do completion on this, that we get a strong type for what the amino acid A actually is and that means we can get strong completion on all the properties of the amino acid known to the information service. And what strong typing gives us is not just the usual properties of strong typing which is like code correctness, for example if I make a mistake here about misspelling amino acids up here then I'll get an error telling me that table doesn't exist and that's all the good things that come from strong typing, but it also gives us strong tooling in that when as we come along and we look at something like the, well there's a good one for instance, so we look at the CAS ID here, we actually see this field is deprecated, we get strong information coming through our compile time telling us that the information service is changing in some way and that we might need to go and use some other property. Okay, so all the usual benefits of strong typing, but in this context where we're working with that sea of information on the outside, as long as schema information can be flowed into the programming language we can make use of it. Okay, right. Okay, so what we've done there, we've integrated all of freebase.com through using a simple change to the programming language which is to use a provider model for libraries, an on-demand compile time computed metadata projection from the information services into the type system of the host programming language. Okay, so let's take a look at a second example and then I'll dig into more about how this actually works in practice. Okay, this one is going to be looking at the World Bank here. So the World Bank is very typical of the information revolution and organization whose mission is primarily about helping the developing world through finance and funding also becomes a data organization. And one part of their mission is to collect and provide data about, data time series information sets about various indicators about countries and regions about the world. So we can, on their website, we can go down and dig in to look for instance at Australia and you can find out all about, all of a range of different development indicators about each of these different countries. You can see the huge number of indicators that are available for access. And what we'd like is all of this to be sort of automatically projected into the programming language for us to work with. And that's what this provider does here. It takes all of that metadata and makes it available as F-sharp, as to the F-sharp system, the F-sharp compiler development environment tools in a strongly type form. And this provider, the source code is available and I'll give this reference again. This is actually available now in the F-sharp.data library. It's open source library on GitHub. I'll give a reference to that later. Some of the namespaces will have changed. This is actually part of the F-sharp.data.dll library now. You can go and look at the source code for this provider. It's only about 500 lines of code or something to do the projection from that information service into the, into F-sharp. And it's a lovely little sample of a standalone type provider. Okay, so once we have the reference to the library, we can get a data context through to the World Bank. And now we have all of that information in the World Bank available. We can go through to individual countries like we're doing here. We can choose Greece, for example. And one of the things we get is the indicator set that's available for Greece. And all of those 4,000 or so different indicators coming from the World Bank as time series data sets are now available for us to use directly in the programming language. So we're interested, for instance, in debt in Greece. I can't imagine why we'd be interested, but maybe we would. You can see there's actually just an incredible number of different kinds of debt in the world. It's just really, it's just, it goes on and on and on. So, but whichever ones you're interested, you can find the time series data sets here. Once you get the, it's a time series here, so it has, you can get either the values associated that or the dates and times associated with the entries in the data set. So let's take a look at some of the data now. Okay, for instance, we'll get a set of countries here. And we'll do a combined chart for the population statistics for those countries. Okay, so you can see how the populations are growing over time for the, with China and India at the top. Okay. Okay, so we could easily just come along and choose a different population statistics. You might be interested in how the youth population is growing as a percentage of the total. We substitute that in there and combine those here. Okay. It's running along and getting the different indicators and you'll see that how that's changing over time as a percentage. And actually the youth percentage is interestingly changing going down in almost all of those countries. Okay. Okay. And a different view of the same thing as a pie chart for the same population data. Okay. Right. Okay. So integrating, so we've seen two examples, integrating all of freebase.com as if it were a library and integrating all of the World Bank's data as if it were a library. Type provider feels just like a library to the programmer. It's a design time component that computes a space of types and methods on demand. It's an adapt, you can think of it as an adapter between the metadata on the outside and the.NET type system on the inside. You can, from a technical point of view, think of it as sort of on demand, scalable, compile time provision of type module definitions. It's a bit, it's a form of compile time metaprogramming but a very simple and sort of straightforward. It's a lot simpler than adding macros to your language. And languages that add macros usually don't do it in an on demand way where the macros are computed lazily, effectively. Haskell, for example, has a macro system but that's, it's funny, it's a lazy language with a strict macro system. F-sharp is a strict language with a lazy macro system. Okay. So on demand types makes your language internet scalable for metadata integration. So how does this work? When we do database, when we, the first line, freebase.getdata context, the compiler or the IDE, it's effectively getting your symbols from the web, from the providers. It asks the provider for about the symbol get data context and the provider says, okay, it's got a return type of freebase data context, whatever that is. Then the compiler sees the code data. Then the compiler IDE will request the contents of the type freebase data context. Well the provider, the component then has to go and ask the freebase metadata service for what's the contents of this type. And then the provider reports the top level domains for freebase and one of them is society, for example. The compiler sees this code and it asks for more information about society and on and on it goes. Okay. So the symbols and the specifications are being provided on demand and everything is on demand right down to the documentation. You don't have to download, you're only accessing the parts of the metadata information service that you actually need to compile your program, which is what the magic that makes it work. You're not downloading all of the freebase metadata just to compile a three line program. Okay. The next theme that comes out of this is that one data source and one mechanism lets you, it's just got an astonishing number of applications. And the way we like to say this in the F-sharp communities, like all your type are belong to us, that any notion of type that you have in the outside world where you can actually both access it at compile time and access the corresponding data at run time, that can be projected into the F-sharp programming language, which makes a whole world sort of part of the F-sharp exosystem now. So SQL is an obvious example, good old SQL programming. The F-sharp, the Microsoft.fsharp, the data that type providers contains this SQL data connection type provider where you just give a connection string and you get link style programming over the contents of the database straightaway. You can, for instance, use this to mash up several different databases. You can take, it's fantastic for data mashup work because your one script can access many different data sources. Instead of in Visual Studio having to go right click, add data reference, add service reference, build a project and kind of have a big lot of files and generated code, all part of your environment, you've just got a 20 line or 50 line script integrating and incorporating all the different data sources. So one, so more routine example is CSV file data. And you can look at examples of this on the TryF-sharp website. You can actually use type providers in the browser through tryf-sharp.org where you can do some F-sharp programming directly from the browser and that includes type providers in F-sharp 3.0. And this is great. You just point it at a CSV file here. Most CSV files come with a header with some metadata and it actually just infers the types from the actual data in the CSV file. It downloads the prefix of the CSV file, looks through the first number of rows, probably defaults to 10 or so and you ask it to infer the types and ignore any errors coming through. And when you get most, since most open government data sets are actually distributed as CSV files these days, then this lets you integrate external data sets and gives you the same strong type in completion over the contents of those data sets as you can see here. So JSON data, this is another interesting way of getting type specifications. And a typical way of using this is you want to access one of those APIs at the start of the talk. But the API documentation just shows you the JSON data that comes back from the API. But you want that in a strongly typed form. And the typical.NET, you know, Java style as you go to then design a lot of classes and use some serialization framework for mapping these in. And it's kind of clunky. And instead, if the F sharp model, you just use a JSON provider and just give it a sample of the data that comes back. And it's usually a sample taken either directly from a documentation or a test call to the service. And that sample is enough to drive a strong typing for the data. So here's the compile time, you can see the angle brackets, the compile time sample that we're giving, which describes the types by sample. And then once we've got that, we get a parser for those types where we can actually use that to access other data here. And I'll just show you an example of doing that with a Twitter feed here. So here we, what have I done here? Okay, I'll move this up. Okay, authenticate us here. I think I need to copy this in. Maybe I've already, I'll do it anyway, 2559. Okay, so there are actually some nice wrappers for the Twitter API in.NET, but not for all of the Twitter APIs. So for instance, this part to get user suggestions for the active user doesn't have a nice wrapper in the standard Twitter API. So we actually can get a strongly typed wrapper for that very, very simply by downloading some sample data here for the data returned from the API. And we then put that sample data into the JSON type provider as a description of the kind of thing we get back from the API. And now when we actually get the suggestions here, and this could, this same code will work for tomorrow's suggestions and any future data returned from that API because the API doesn't change because it's got a version number 1.1 here which will keep it steady. And then now as we iterate the suggestions here, then we can see the data coming back, the name, the size, and the slug here, whatever the slug means associated with the data. So we have a strongly typed view over the data coming back from the API. A slightly more complex piece of data coming back is this trends data. And we can get the same thing here. We have the trends data. We get a strongly typed view over the trends data. And we can now, so the key thing is here is that at these kind of points we get strong typing here. So we can see that the data. And I haven't even looked closely at this data and I'm already navigating it and noticing that it's got locations information, so for trending locations around the world. And I didn't know that until the strong typing sort of guides us through the process of using it. And so the contents of the data as well, you can see, you can tell whether it's promoted data is associated with events and so on the URL. And we can iterate that data here. Okay, so you can see all the various trends right now coming from Turkey, for example, here. Okay, so types from JSON data, it just makes working with JSON so simple in a strongly typed setting and so sort of frictionless, right? There's no need to be thinking about building object models corresponding to this data. You can do the same thing with XML, the same, exactly the same model, you give it a sample and you get parsers and serializers and the like for working with that data, return data. There are schematized protocols like OData where you just point the type provider at the service here. These angle brackets are saying give me the types and information catalog coming from the OData Netflix service. OData is a protocol supported by many Microsoft products, so you can just point it at the end point for SharePoint OData services or Excel or other products, for example. There are samples around for doing this with Hadoop and Hive. This lets you just point at the head node here of a Hadoop cluster. You give it the port to access on and you access through to the metadata associated with the tables, the distributed file system tables stored in the Hadoop system. Again, it just makes working with Hadoop substantially, with HDFS and Hadoop substantially easier. World Bank, we've seen this one. You can integrate whole data markets. You could imagine doing this to all of the Bloomberg or the Reuters catalog of information, for example, and integrating all of that into a strongly type setting. Old-fashioned protocols like Wizzle, WMI data on the local machine is another massive piece of schematized information that could be integrated in. Every base we've seen. So another really, really interesting use of this mechanism is to access from F-sharp into another programming language. So TypeScript is a strong typing or sort of pseudo typing over the top of JavaScript. And part of what TypeScript can consume are these.ts files, which give strong typings over existing JavaScript libraries, kind of like header files for JavaScript libraries. And you can, when you're compiling F-sharp down to JavaScript, using, in this case, using something called the FunScript sort of dynamic, the FunScript compiler for F-sharp code, then it's useful to be able to just access all of those TypeScript, all of those JavaScript libraries through a strong typing. And so you do that with the TypeScript type provider in F-sharp. You're in F-sharp here and you point it at a.ts file. And then you, this is for jQuery and you can do things like j, we've got the j here, so you can do things like j.jQuery over here. If you're interested in looking at that, there's, so let's take a look at FunScript here. This is the home of the FunScript project. It's again a fully open source project part of the F-sharp open source community. You can see some great examples of how to use F-sharp to write fully JavaScript HTML5 apps. And you're using Type providers and Type providers through to the World Bank and movie databases and all sorts of other things purely from F-sharp code and sitting on top of TypeScript typings. You can see the various TypeScript typings that are sitting on top of here, Google Maps or jQuery and Node and so on. Okay. So FunScript is a great thing to get involved with. There's another F-sharp to JavaScript system, a professional system called Web-sharp. It's also open source. And that also I believe has a TypeScript type provider as part of it. And that's a fabulous thing to get involved with as well. That's this Web-sharp system here. Okay. All right. So the last thing we'll look at is integrating another programming language which is R, the R statistical programming environment. And I mean, F-sharp has some nice visualization libraries with F-sharp.charting. But R has lots and lots of statistical functionality and visualization functionality which we'd like to be able to access from F-sharp. And so we can do that with the R-type provider. So let's take a look at what that looks like here. So that's the R-type provider. Again, an open source component done by Hedge Fund called Blue Mountain Capital in New York. And when you reference the R-type provider, then any R packages you've got installed on your machine become available under this R-provider.namespace environment. Okay. So all of these different packages which are the base set of packages that come with R are now available for use from the F-sharp language. Actually, you get almost better tooling than you get in R in some ways. There's a little bit of friction at the boundary because R is basically an untyped system, a dynamically-typed system. But there's enough metadata there to allow us to do some kind of interop through to the system. So we can, let's do, use this to, I think, let's check that I have, okay, let's do this again. Okay. So this is downloading a set of stock prices here and then using this R data frame library here to build a data frame, a cube of the relationship between the labels and the stock prices. And then using this sort of standard piece of R visualization machinery pairs to look at the pairwise correlations between the different stock prices. And you can work out various statistical things about how, for instance, Microsoft and Apple correlate here. Okay. Okay. So F-sharp to R. Okay. So a couple of other themes in this work which I find personally really intriguing. One is this thing about working with these external data sources at multiple scales of, of, of schematization. Data scripters need to work with lots of different scales or granularities of schematization. They might need to work with just all the entities in the database to search all the entities. For instance, show me any entity which has got anything to do with New York. Okay. That's one model. Another model is to dig down into sort of the organizing things by type. So just show me all the different models of, of, of cars, for example, the table of models of cars. It's also quite interesting to go dig down to a level to give each individual its own unique type or own unique name. Okay. And the, we can see that, for example. Let's take a look at what that looks like in free base. First, have a think about like if you've got six objects, right, then there's, and there are three of them have type A and three of them have type B. Then there's three different ways you can present the API. You could just have a, a list of objects. You could have organized by A and B. Or you could name each individual A, A1, A2, A3, and AB1, B2, B3. So now when you want to access, let me just load this up here. Okay. Load up free base again. Okay. Now, if you want to, let's have one more. This one here. Okay. If I want to find hydrogen, I can do it by name. Or I can actually come along to the individuals property in the chemical elements here. Chemical elements start individuals. And now I've got a strong name through to each of the individual entities in this database. And remember there are like six, there are like 50 million something entities in the database altogether. And each of those can be given a strong name. By default, we actually just take the first 1000 from each of the, each of the tables here. So you're just getting the first 1000 individuals or there's, there's, that come through from the system. And there's less than 1000 chemical elements. So that's all of the chemical elements. So we can search through those, for example. Okay. So you can do this with just about everything, with every table automatically. So if you're interested in a type of getting the, of accessing through to particular infectious diseases, you can do that here. If you're interested in accessing individual computer scientists, you can do that here and work with those. Don't tell the NSA about this feature. Okay. Really, really no one. They already know. Sorry. Okay. Okay. You can see that, you can see the interesting thing about this actually is in the context of programming with real world information is these things don't actually disappear. You know, these things are around for forever. So if we come along to arts and entertainment and we're interested in opera, for example, and we're interested in operas here, we come along to individuals. It's the first time I've looked at this one. So just takes a moment to download the set on the first time. Okay. Then you can see the set of operas coming through here. Okay. Okay. So that intrigues me because what you see people, as people are working with these information sets, they start off trying to generalize an algorithm and then something goes wrong and then they need some sample data to work with them. They need to dig into one or two individuals in particular. Then they come back and they work at the more general level again. And so you need, you can give a unique type to every individual in the database if you want. And so you need a language that can scale to support massive amounts of scalable metadata to operate at all the different levels. Okay. The next theme I'll bring out in the last couple of minutes is about units of measure. These information services often come with units of measure. For instance, the highest wins is in kilometers per hour. As that information gets transformed into F-sharp, it actually does a normalization into the SI unit system because F-sharp likes the SI units. So it actually changes it into meters per second here. So you actually get that metadata being propagated into the type system of the units of F-sharp so that you know that this top win rating of the Hurricane Sandy here is actually in meters per second. Okay. A word about schema change. Everyone always asks about this. What happens when types change on the outside? The more granular you are, the more fine-grained you are in your projection of metadata, then the more you're exposed to schema change. But many data sources are really surprisingly stable. In some ways, I think these data sources are sometimes more stable than our programming frameworks. You get more churn at conferences like this of new programming frameworks and new ways of doing things, then you get change in some of these data sources. Some data sources provide these snapshot dates that let you snapshot either the metadata or the actual data at a particular point in time, sort of versioning systems, and you can project those across into the versioning mechanisms of the programming language. If change happens while you're working with a data source, the provider can give an invalidation signal while you're working with it. Just to mention that if you're erasing types, so all these types disappear at runtime through erasure, and that makes your compile code much less fragile to change. Okay. If you want to use some of these type providers, I've mentioned fsharp.data. It's an open source library. You can get it on Nougat. Just go to fsharpgithub.io, fsharpdata, and you can just use it in Visual Studio or on the Mac or Linux or any system that supports fsharp code. Okay. So just to finish off, the big themes are we live in an information revolution. There are many aspects to that, but that needs to have ramifications for how we think about programming languages, how we design our languages, and how we think about information access. We need to bring information into the languages at internet scale, and I think we've broken some new ground in how this can be done in practical settings. If you'd like to find out more, go to fsharp.org. You can try it online in your browsers right now on trifsharp.org, and there's great documentation on the MSDN Visual Fsharp pages as well. And thank you very much, and I think we've got a few minutes for questions as well. Thanks. Questions? Yeah. Yes. Yes. So a part of every type provider is to provide a query implementation, an Iquirable link implementation effectively here. So when we're accessing the books database, it's got three million books in it, so we can't possibly bring that table to the client side. So this is what an fsharp, this part here is what an fsharp link query looks like. Just says the obvious thing. We're using a freebase operator called approximate matching, which is a bit looser than exact matching in.NET, and we're just going to take the top 10 books, and then we can impact that into the show grid function, and that gives us the books here on the right. Okay. So querying is absolutely essential to nearly all of these remote information services. Yeah. Yeah. So we haven't done anything with that particular setting. It's going to work just fine. Your metadata comes across, you can project that into types, and there's no performance overhead to any of the metadata integration that we do, because that's a compile time only feature. The actual runtime behavior is just like it normally would be in any way of accessing through to the system. Okay. So there's zero compile time overhead to this mechanism unless you've made a mistake in your code and how you're doing things. Yeah. Any more questions? Okay. Then I think we're done. Thanks very much.
 Hi everyone.  Great to be back in Oslo.  It's, I came here for a Java conference last time and it's NDC this time.  So my name is Don Sein.  I'm a principal researcher at Microsoft Research in Cambridge.  I, my mate, I guess some people know me as having worked on C-Sharp and the design of  the F-Sharp language.  I, F-Sharp, I, as one of the things I'll be bringing out about the F-Sharp programming  language today is that one of the exciting things that's going on is it's a strong open  source community and I see myself as a contributor to that community rather than as a sort of,  sort of leader of that community.  And just one of the, one of the many people contributing.  And so I prefer to introduce myself these days as a F-Sharp contributor who happens  to also be a principal researcher at Microsoft Research.  So today I'm going to be talking about what for me has been perhaps the most exciting,  the most inspiring, the most intriguing programming language feature I've had the pleasure to  work on during my time at Microsoft Research.  And it's all about the fundamental problem of how we bring data, information, services  and programming together and do this at a scale that's suitable for the modern era.  And you can use it today in product form in Visual Studio 2012 in F-Sharp 3.0.  You can use it through the open source editions of F-Sharp on all sorts of platforms to access  all sorts of information services in many, many different kinds of programming settings.  But beyond that, it's actual existence in F-Sharp today.  I hope it also serves as an inspiration for what programming can be in the future for  a whole range of different languages.  We've seen this work have, I know through personal contacts and emails, having direct  and strong influence on the designers of C-Sharp and the designers of OCaml and of Scala.  And I believe through that and through, as it flows through the industry, we'll see this  gradually roll out in many different forms and perhaps many of which we might forget  the some of the background that comes from here.  But that's sort of the role of research is to come up with ideas which spread of their  own accord through many different settings in the industry.  So I'm excited from both from a research perspective and also extremely happy with what we've  made available to you as developers in F-Sharp 3.0 both in Visual Studio and in the open  source editions of F-Sharp.  So the fundamental problem as we all know is the world is information rich.  It's actually a very, very interesting week to give this line about information richness.  And it's, I mean, we know the many, many positive sides of information richness and  we know some of the negatives that that might bring and some of the challenges that brings  to our society.  The way I like to put it is, I mean, as we've known for a long time, the web changes absolutely  everything.  The web changes the whole way we work as individuals.  It enables all sorts of experiences that with using devices that we never thought would  be possible.  And it also, it changes politics, changes communications, changes many, many things.  But this talk is really about how it can go down to the very base level and change some  assumptions we make about programming languages and about how we think about the role of programming  languages in the world, especially strongly typed programming languages.  And the context is the information revolution.  This is just one of many ways of visualizing this.  This is taken from programmableweb.com about the rise of open APIs on the web.  You can see the numbers below here.  There's basically an exponential rise in the sort of digital API context that's available  on the open web.  And we don't expect this kind of thing to stop anytime soon.  The size, if you like, the sort of, you know, the magnitude of the information context around  us will just continue to grow and grow.  And this is the world we have to interact with as programmers.  So modern programming is intensely information rich.  You see manifestations of that in data science.  You see it in programming for devices in that these are now connected always, nearly always  connected through to some kind of set of information sources.  You see it in the enterprise for a long time.  The enterprise has sort of been a microcosm of what is now happening sort of universally  across programming.  But our languages are information sparse.  They don't in themselves contain any information.  They don't actually contain for our strongly typed languages.  They don't contain representations of information coming from the outside.  You won't in a C-sharp library space, you won't find a definition of a chemical element  or an amino acid or a football player or any or genome or any of the information sources  that people actually have to deal with every day.  So instead, our languages contain these very generic information adapters like XML or text  or bytes or JSON sort of APIs to help us do sort of untyped generic manipulation of data.  And that's great for many purposes.  But for many purposes, we need to program against precise data structures.  And those need to come from somewhere and be brought into the language from somewhere.  So the developer's perspective is in some sense the languages do not integrate information  well.  If they do, it's sort of either non-intuitive or it's not simple or disorganized or static  or high friction.  We have to use these generic adapters like web requests or XML readers or JSON.  And this is a problem.  It's a problem especially for strongly typed languages.  Strongly typed languages are incredibly efficient information manipulation devices.  Once you have data under the type system of a modern strongly typed language, especially  a strongly typed functional programming language, or one with the sort of expressive power of  say the.NET and C-Sharp type system, then that's great.  Everything kind of goes pretty well.  There's still problems.  Nulls, for instance, can be a problem in some type systems.  But once you have it under the type system, things are good.  But if the cost of getting information under the discipline of the type system is too high,  then any efficiency you gain by getting information under types is just lost around the edges.  And this is one of the main reasons for the big resurgence of interest in dynamically typed  programming languages is because you need to interoperate with the outside world more  than you need to process things internally in the language.  So we need to bring information into the language, integrated into the language at internet scale,  strongly tooled and strongly typed.  And that's what we're trying to do here.  But back in the good old days, the world was neatly separated into statically typed and  dynamically typed languages.  And everything was simple and the textbooks all were nice and clear.  And this has sort of changed a lot because of this problem of integrating with the external  world and also the rise of a lot of different kinds of programming as well.  So we now see a lot more sort of overlap.  And I guess there's a search on in the middle to make statically typed languages more dynamic.  You can see that same C sharp 4.0 to make dynamically typed languages more static through, for instance,  lots of type inference in JavaScript implementations or to put sort of static facades over dynamically  typed languages like in TypeScript.  And then there's sort of moderate static typing, some of the guarantees that you get from static  typing in limited ways, but expand the use of static typing.  And that's what we're doing in this talk here.  But before we get into that, I need to tell you about the context that we're doing it  in, which is F sharp.  And let me, so I'll just run through a little bit of material about F sharp.  I'll do this fairly quickly, but just to give you a locator for where we are with F sharp.  So these days, I like to separate between two things.  One is F sharp as a language and one, the other is a visual F sharp tool.  So the visual F sharp tools are part of Visual Studio.  F sharp as a language is still designed primarily at, through, at Microsoft and the compiler  implementation is done at Microsoft.  But F sharp as a language is open source and is cross platform.  And this has been a really, I mean, very exciting developments in the F sharp community through  the development, through the community of this F sharp software foundation.  And this is a software foundation in many ways set up to be orthogonal, to give the  reality to F sharp being open source.  And F sharp, they see it as their mission to sort of complement what's done by the core  language and compiler team at Microsoft and to bring F sharp onto Mac, Linux, Android,  iPhone, HTML5 and, and platforms like FreeBSD and even GPGPU programming.  And to give sort of a non, I think a non-Microsoft perspective on, on F sharp.  I'll talk a bit more about that in a bit.  But just to get across that, I think it's worth separating out those two ideas.  Visual F sharp tools are part of Visual Studio.  F sharp as a language is open source and cross platform.  So our mission statement sort of F sharp is strongly typed data immersive functional first  language for simple code.  We used to say simple code for complex problems.  Okay.  So actually it's just about simple code when you really kind of come down to it.  I mean, you happen to be able to solve a range of complex problems with it.  I'll be talking, I'll be giving another talk here today at, just after lunch, talking about  functional first programming more broadly and how, how you can succeed with functional  first programming in, in industry.  And I'll dig into this a bit more of what sort of roles it can play.  But code simplicity is absolutely at the core of a lot of what we do in strongly typed  functional languages and in F sharp in particular.  So there's a function to swap two elements of a tuple and in C sharp that's okay.  You can write this in C sharp, but it just kind of gets a bit longer and you take the  rotations of a tuple XYZ and you get three, three rotations and yeah, it's fine.  But you know, what do you do in C sharp?  It's like, it's actually annoyingly hard to write out this kind of simple data manipulation  lists and tuples.  And it's just one example of many, many examples of F sharp coding which drop out as sort of  more simple, more declarative, better properties in terms of robustness and particularly because  of the lack of nulls in, in, in F sharp coding.  And there's another one reducing FXYZ.  You can do the same sort of thing in C sharp.  But again, it's just a bit, bit longer and not as declarative.  Simple code defining a type, a class in F sharp of one line, it might go on for many  lines in C sharp.  And you can see that in our data access work which will enable us later on in this talk  to write things like this.  Get me the, get me the population age of zero to 14 is a percentage of total of Brazil.  You know, it kind of doesn't get more simple than that.  We're integrating this information from the World Bank Information Service.  And it's just about this, it's sort of bringing out that what we're doing in this talk is  following on with the general theme.  The general mission statement of F sharp about being about simple code carries through to  simplicity in data access as well.  One thing I'll, I, I, I like to, to, to bring up at this point is why you might not want  to use a functional programming language.  And this is actually a very humorous piece of satire about, about why you, why you might  not want to use a strongly type functional language.  Like if you get paid by the line, for example, that's a, you know, then you, you might prefer  to write this sort of code.  I compare it, you know, with a couple of lines of F sharp code and so on.  Love me some curly braces.  I can't do without them.  Okay.  So if you're that kind of program, if you really just can't do without your curly braces,  then there, you know, it might not be the right setting for you, but you know, you probably,  I reckon you probably can live without those, those curly braces and you can work on that.  Okay.  How does F sharp get used in practice?  I'll, I'll be talking more about this in my other talks.  So just a couple of quick examples.  One is, for example, application to balance the national power generation schedule for  a power company in the UK.  It's very common that F sharp gets used as a, as a component inside a larger dot net  or maybe a dot net service with a JavaScript HTML5 front end or just within a larger dot  net application.  One of the interesting things about F sharp is that, is that property that because it's  so easy to interrupt not only from F sharp to the rest, for F sharp to access dot net  libraries and data services as we'll see in this talk, but you can interoperate from dot  net to F sharp very easily, which means you can encapsulate your use of F sharp into however  many components you happen to be sort of right for you to use.  And that tends to start off as one component and then people expand out their use of F  sharp.  So Kaggle, for example, Kaggle is a data science company.  They run competitions for a sort of crowdsourcing of data analysis.  You can enter these competitions and try and do the better data and statistical analysis  than other people over these data sets.  Kaggle themselves who run these competitions use F sharp for their core data analysis algorithms.  And then they, as I said, you start off with some core data analysis algorithms and then  you move more and more of your application into F sharp.  And the usual things that people report, and this has been a long standing thing with strongly  type functional programming, is fewer bugs, easy to refactor, easy to read, and stronger,  and because the strong typing gives the better robustness and fewer bugs.  Okay.  So as I said, F sharp is changing a bit.  There was this view, I think, before the F sharp was for Windows.  That's just not the case anymore.  F sharp runs on many platforms.  There's also, I think, a view of Microsoft make F sharp, and that's actually changed a  lot.  F sharp has many contributors now.  The core compiler language is still done by Microsoft, the team at Microsoft, but for  Windows and that gets taken and the other contributors take it to other platforms.  I think there was a, when we came out with F sharp, when Microsoft come out with something,  we have a bit of an overly strong, a bit heavy, I think.  So we came out with F sharp from Microsoft's perspective, and that's actually changing  to be many different perspectives on what the programming language is for through F  sharp.org.  And yeah, I guess it's sort of a, it's instead of seeing F sharp as being something that  kind of comes from Microsoft for Microsoft's purposes, there's now many different contributors,  some of which are Microsoft, some of which are Microsoft research, and in fact Microsoft  research have somewhat different goals than Microsoft in contributing to F sharp.  Our role is to partly to take programming forward through a language like F sharp.  We have everything flowing in from Nougat into the F sharp environment.  You can access all the Nougat packages, 11,000 of them from F sharp.  There's lots of good community sites.  We have Xamarin is a company which provides F sharp support on the iOS and Android platforms  along with them primarily C sharp support, but an F sharp option as well, and quite  a few startups contributing to F sharp as well.  Just to give an example of the exo system around F sharp, that is, not these aren't  intrinsically F sharp packages, these are.NET packages, and one of the things about F sharp  is it kind of lives in this exo system, is embedded in this exo system of.NET.  If you want to access Amazon web service,.NET SDKs, you can do that, and you can access  all the great services from Amazon.  If you want to access the Azure.NET SDKs, you can use those from F sharp, and they're  all open source as well, and you can go and do your cloud computing with either of these  platforms.  You want to learn more about particular areas of F sharp, data science, machine learning,  cloud or data access and so on.  Go to F sharp.org, there's lots of good information.  You can contribute to those pages.  You just send a pull request on the GitHub source for these pages, and then the community  organization will integrate your information into that setting as well, so you can get  involved with the community foundation.  Okay, so back to the main topic, bringing information into the language, at internet  scale, strongly tooled and strongly typed.  Okay.  Right, so let's start with a small piece of information, which is freebase.com.  So freebase is an entity graph.  You can more or less assume this is taking all of Wikipedia and doing entity extraction,  looking at those structured panels of information that are down the right, for instance, that  are given organized view over, say, all the different individual entities, individual  pages in Wikipedia and actually elsewhere as well, and integrating those into a single  web database slash ontology slash service, whatever you want to call it.  It's a pre-base, has been acquired by Google and it's a predecessor to what's called the  Google Knowledge Graph, which is sort of an even bigger version of freebase.  Freebase has got a publicly accessible API, which is what we'll be using.  The Google Knowledge Graph, as far as I know, doesn't have that, but it is much bigger.  But it's essentially, again, everything that Google sees on the open web run through a  set of entity extraction algorithms and graph database wizardry and a whole lot of other  amazing stuff that I don't know much about.  The end result is an extraordinary sort of source of organized, schematized information  about the real world.  For instance, you want three million books that you can query and search, then this is  a good place to find them.  If you want whole databases of TV and film and music and sports and government and just  everything, you can find thousands and millions of topics under here.  A lot of people, a lot of different companies are getting into entity extraction.  This information is useful for all sorts of things.  One of the things it's mainly used for is search to provide those lovely little structured  panels you get either from Bing or from Google when you type in something into the search  engines.  They kind of use it to give a more entity-based structured view of the world around us.  But wouldn't it be nice if we could just access all of this information from the programming  perspective and do it in a sort of strongly typed and organized way?  So we might, for example, be trying to do something like, where's my talk gone?  Okay, now I've got to skip through.  We might be after something like accessing the chemical elements as part of a chemistry  app for a phone or for a web or for an educational application.  And we'd like to access some basic information about chemistry.  Okay, before I begin, I'll just do a little bit of F-sharp programming here.  What I'm using here is F-sharp interactive in Visual Studio.  I'll just put this down the bottom here.  I've created myself an F-sharp script and I've loaded up some scripts up above.  The most important one of which here is this one which is called show, which provides me  with a function called show, which allows me to take some data.  I've created here the numbers from 0 to 99 and I can call the show function on it and  display the data over in the right-hand window.  I can also do the same thing by piping, using this piping syntax, if I change that to 59  and then show the numbers over on the right.  Now, I'm taking the numbers and I'm piping them into the show function here.  You'll see me use that quite a lot.  I can also use this show grid function to show a more sort of tabular display.  And for instance, if I get the table of squares and I do a show grid over on the right, you'll  see the values coming out.  For some reason, it's showing duplicates when I'm not sure why that is.  That's a bug in the show function, in the show grid.  I'll also do some charting, so we'll take that data of the table of squares.  So this is the numbers for i from 0 to 99 generating i and the square of i.  And that's our data there, that's our data set.  And we can pass that data set to the charting library, which will then just display the  data set in the obvious way.  Get rid of this.  So let's take a random point cloud, for example, so 10,000 points where we're multiplying two  random numbers here, across here, and you see them drift down to an average of about  a quarter down here.  Okay, just to show how we might do interactive data work with F-sharp, in this one we're  going to take a stream of events coming from this window over here.  And as the stream comes in, we're going to collect up a data set and display that data  set in the chart over on the left, so that as I move my mouse over on the right, we're  accumulating a data set from the events on the outside and displaying the data set over  on the left.  Okay, so that was to take the mouse move event to extract out the x and the y coordinates  from the mouse move and to create an updating chart from updating chart.line.  This is using, if you're wondering, this is using the F-sharp dot charting library, which  is, and in the latest edition of F-sharp charting, you'll see that updating chart has actually  been renamed to live chart if you're going to try and do this at home.  Okay, so to take another one here, we're going to take, again, showing how functional programming  plays with working with data and information.  We're going to take the mouse move events from over here.  We're going to extract the y coordinates.  We're going to sample them every 30 milliseconds.  We're going to take a 3,000 millisecond, three-second interview, interval over that, and collect  up that data set, updating data set, and we'll have that displayed over here, and you can  see our data set has been generated.  And I'd like you to think, how long would it take you to do that kind of data specification  and transformation and visualization in an object-oriented kind of way?  Okay.  So this is fundamentally, you want combinators like sampling, you want them like taking a  window, that's exactly what you think about when specifying the problem is, I want to  extract out the y coordinates, I want to sample, I want a window time interval, and I want  an updating chart of that data set.  Right, but let's look at information integration.  Okay, so what we've done with F-sharp 3.0 is really, part of the magic of it is that  it's just such a simple change to the language, okay, which is that instead of referencing  a library, we also use to the idea of referencing a DLL, okay, we know that in.NET, or.NET  DLL contains a set of types, a set of classes, a set of namespaces in classes, and we know  that in the Java context, you reference a JAR file, it's the same thing, it contains  some types, okay.  So why not move to a provider model for libraries instead of a fixed set of types?  So instead of referencing just a finite collection of types, we're actually referencing a component  which computes types on, types and the contents of types on demand, okay.  So we'll go through what that means in more detail in a bit, but effectively it's just  doing what we often do in computing, which is when we have something, we turn to use  a provider model for that something instead.  So when I'm referencing Freebase here, that Freebase.dll only contains one or two types,  okay, but those are special components which can compute types on demand, okay, and compute  them at compile time and compute them in the development environment.  This DLL acts as a plug-in to Visual Studio and it also acts as a plug-in to my execution  environment down below.  So we're moving to a provider model for types and that DLL here lets us have this Freebase  data provider and we give it some static parameters.  We tell it how we're going to access the service, so give it the API key to our service, which  is known at compile time to access the metadata for Freebase and we give it another parameter  which I'll explain more in a bit.  And once we have that executing the lines through to this data line, then the type of  this data object is now represents the entire universe of information that's available through  Freebase, okay.  So that's effectively handled to a structured type view of everything on Wikipedia.  So that means when we start to navigate the properties of this data object, we're actually  sort of navigating not at runtime where we were actually accessing the data, but at compile  time where navigating the metadata associated with this service.  Okay, so in many cases I haven't actually previously accessed these, I've never looked  at that engineering domain before, okay.  So you're interested in knowing things about engineering and you're knowing, for instance,  what different kinds of batteries are available in the world, then we can at compile time  get strong typing and strong tooling to access all the different parts of this information  service.  So for instance astronomy, we can come through, we can come through to astronomers or to stars  here.  One moment.  Okay, to stars or star systems and all the different tables of information available  in the information service about astronomy.  So let's take a look at some of the actual data, for instance, we can come through to  biology and we can look at the amino acids.  The double ticks used in F-sharp are just because the name contains a space which is  very common in this kind of working.  So let's get the table of amino acids and we have it available over on the right here.  Okay, so now this is strongly typed data, although it might look like it is a sort of  a dynamic language feature, so if I actually iterate the amino acids here and I look and  I try and do completion, we haven't actually iterated these, I'm trying to do completion  on this, that we get a strong type for what the amino acid A actually is and that means  we can get strong completion on all the properties of the amino acid known to the information  service.  And what strong typing gives us is not just the usual properties of strong typing which  is like code correctness, for example if I make a mistake here about misspelling amino  acids up here then I'll get an error telling me that table doesn't exist and that's all  the good things that come from strong typing, but it also gives us strong tooling in that  when as we come along and we look at something like the, well there's a good one for instance,  so we look at the CAS ID here, we actually see this field is deprecated, we get strong  information coming through our compile time telling us that the information service is  changing in some way and that we might need to go and use some other property.  Okay, so all the usual benefits of strong typing, but in this context where we're working with  that sea of information on the outside, as long as schema information can be flowed into  the programming language we can make use of it.  Okay, right.  Okay, so what we've done there, we've integrated all of freebase.com through using a simple  change to the programming language which is to use a provider model for libraries, an  on-demand compile time computed metadata projection from the information services into the type  system of the host programming language.  Okay, so let's take a look at a second example and then I'll dig into more about how this  actually works in practice.  Okay, this one is going to be looking at the World Bank here.  So the World Bank is very typical of the information revolution and organization whose mission  is primarily about helping the developing world through finance and funding also becomes  a data organization.  And one part of their mission is to collect and provide data about, data time series information  sets about various indicators about countries and regions about the world.  So we can, on their website, we can go down and dig in to look for instance at Australia  and you can find out all about, all of a range of different development indicators about  each of these different countries.  You can see the huge number of indicators that are available for access.  And what we'd like is all of this to be sort of automatically projected into the programming  language for us to work with.  And that's what this provider does here.  It takes all of that metadata and makes it available as F-sharp, as to the F-sharp system,  the F-sharp compiler development environment tools in a strongly type form.  And this provider, the source code is available and I'll give this reference again.  This is actually available now in the F-sharp.data library.  It's open source library on GitHub.  I'll give a reference to that later.  Some of the namespaces will have changed.  This is actually part of the F-sharp.data.dll library now.  You can go and look at the source code for this provider.  It's only about 500 lines of code or something to do the projection from that information  service into the, into F-sharp.  And it's a lovely little sample of a standalone type provider.  Okay, so once we have the reference to the library, we can get a data context through  to the World Bank.  And now we have all of that information in the World Bank available.  We can go through to individual countries like we're doing here.  We can choose Greece, for example.  And one of the things we get is the indicator set that's available for Greece.  And all of those 4,000 or so different indicators coming from the World Bank as time series  data sets are now available for us to use directly in the programming language.  So we're interested, for instance, in debt in Greece.  I can't imagine why we'd be interested, but maybe we would.  You can see there's actually just an incredible number of different kinds of debt in the world.  It's just really, it's just, it goes on and on and on.  So, but whichever ones you're interested, you can find the time series data sets here.  Once you get the, it's a time series here, so it has, you can get either the values associated  that or the dates and times associated with the entries in the data set.  So let's take a look at some of the data now.  Okay, for instance, we'll get a set of countries here.  And we'll do a combined chart for the population statistics for those countries.  Okay, so you can see how the populations are growing over time for the, with China and India  at the top.  Okay.  Okay, so we could easily just come along and choose a different population statistics.  You might be interested in how the youth population is growing as a percentage of the total.  We substitute that in there and combine those here.  Okay.  It's running along and getting the different indicators and you'll see that how that's  changing over time as a percentage.  And actually the youth percentage is interestingly changing going down in almost all of those  countries.  Okay.  Okay.  And a different view of the same thing as a pie chart for the same population data.  Okay.  Right.  Okay.  So integrating, so we've seen two examples, integrating all of freebase.com as if it were  a library and integrating all of the World Bank's data as if it were a library.  Type provider feels just like a library to the programmer.  It's a design time component that computes a space of types and methods on demand.  It's an adapt, you can think of it as an adapter between the metadata on the outside  and the.NET type system on the inside.  You can, from a technical point of view, think of it as sort of on demand, scalable,  compile time provision of type module definitions.  It's a bit, it's a form of compile time metaprogramming but a very simple and sort of straightforward.  It's a lot simpler than adding macros to your language.  And languages that add macros usually don't do it in an on demand way where the macros  are computed lazily, effectively.  Haskell, for example, has a macro system but that's, it's funny, it's a lazy language  with a strict macro system.  F-sharp is a strict language with a lazy macro system.  Okay.  So on demand types makes your language internet scalable for metadata integration.  So how does this work?  When we do database, when we, the first line, freebase.getdata context, the compiler or the  IDE, it's effectively getting your symbols from the web, from the providers.  It asks the provider for about the symbol get data context and the provider says, okay,  it's got a return type of freebase data context, whatever that is.  Then the compiler sees the code data.  Then the compiler IDE will request the contents of the type freebase data context.  Well the provider, the component then has to go and ask the freebase metadata service  for what's the contents of this type.  And then the provider reports the top level domains for freebase and one of them is society,  for example.  The compiler sees this code and it asks for more information about society and on and  on it goes.  Okay.  So the symbols and the specifications are being provided on demand and everything is  on demand right down to the documentation.  You don't have to download, you're only accessing the parts of the metadata information service  that you actually need to compile your program, which is what the magic that makes it work.  You're not downloading all of the freebase metadata just to compile a three line program.  Okay.  The next theme that comes out of this is that one data source and one mechanism lets you,  it's just got an astonishing number of applications.  And the way we like to say this in the F-sharp communities, like all your type are belong  to us, that any notion of type that you have in the outside world where you can actually  both access it at compile time and access the corresponding data at run time, that can  be projected into the F-sharp programming language, which makes a whole world sort of  part of the F-sharp exosystem now.  So SQL is an obvious example, good old SQL programming.  The F-sharp, the Microsoft.fsharp, the data that type providers contains this SQL data  connection type provider where you just give a connection string and you get link style  programming over the contents of the database straightaway.  You can, for instance, use this to mash up several different databases.  You can take, it's fantastic for data mashup work because your one script can access many  different data sources.  Instead of in Visual Studio having to go right click, add data reference, add service reference,  build a project and kind of have a big lot of files and generated code, all part of your  environment, you've just got a 20 line or 50 line script integrating and incorporating  all the different data sources.  So one, so more routine example is CSV file data.  And you can look at examples of this on the TryF-sharp website.  You can actually use type providers in the browser through tryf-sharp.org where you can  do some F-sharp programming directly from the browser and that includes type providers  in F-sharp 3.0.  And this is great.  You just point it at a CSV file here.  Most CSV files come with a header with some metadata and it actually just infers the types  from the actual data in the CSV file.  It downloads the prefix of the CSV file, looks through the first number of rows, probably  defaults to 10 or so and you ask it to infer the types and ignore any errors coming through.  And when you get most, since most open government data sets are actually distributed as CSV  files these days, then this lets you integrate external data sets and gives you the same  strong type in completion over the contents of those data sets as you can see here.  So JSON data, this is another interesting way of getting type specifications.  And a typical way of using this is you want to access one of those APIs at the start of  the talk.  But the API documentation just shows you the JSON data that comes back from the API.  But you want that in a strongly typed form.  And the typical.NET, you know, Java style as you go to then design a lot of classes  and use some serialization framework for mapping these in.  And it's kind of clunky.  And instead, if the F sharp model, you just use a JSON provider and just give it a sample  of the data that comes back.  And it's usually a sample taken either directly from a documentation or a test call to the  service.  And that sample is enough to drive a strong typing for the data.  So here's the compile time, you can see the angle brackets, the compile time sample that  we're giving, which describes the types by sample.  And then once we've got that, we get a parser for those types where we can actually use  that to access other data here.  And I'll just show you an example of doing that with a Twitter feed here.  So here we, what have I done here?  Okay, I'll move this up.  Okay, authenticate us here.  I think I need to copy this in.  Maybe I've already, I'll do it anyway, 2559.  Okay, so there are actually some nice wrappers for the Twitter API in.NET, but not for all  of the Twitter APIs.  So for instance, this part to get user suggestions for the active user doesn't have a nice wrapper  in the standard Twitter API.  So we actually can get a strongly typed wrapper for that very, very simply by downloading  some sample data here for the data returned from the API.  And we then put that sample data into the JSON type provider as a description of the  kind of thing we get back from the API.  And now when we actually get the suggestions here, and this could, this same code will  work for tomorrow's suggestions and any future data returned from that API because the API  doesn't change because it's got a version number 1.1 here which will keep it steady.  And then now as we iterate the suggestions here, then we can see the data coming back,  the name, the size, and the slug here, whatever the slug means associated with the data.  So we have a strongly typed view over the data coming back from the API.  A slightly more complex piece of data coming back is this trends data.  And we can get the same thing here.  We have the trends data.  We get a strongly typed view over the trends data.  And we can now, so the key thing is here is that at these kind of points we get strong  typing here.  So we can see that the data.  And I haven't even looked closely at this data and I'm already navigating it and noticing  that it's got locations information, so for trending locations around the world.  And I didn't know that until the strong typing sort of guides us through the process of using  it.  And so the contents of the data as well, you can see, you can tell whether it's promoted  data is associated with events and so on the URL.  And we can iterate that data here.  Okay, so you can see all the various trends right now coming from Turkey, for example,  here.  Okay, so types from JSON data, it just makes working with JSON so simple in a strongly  typed setting and so sort of frictionless, right?  There's no need to be thinking about building object models corresponding to this data.  You can do the same thing with XML, the same, exactly the same model, you give it a sample  and you get parsers and serializers and the like for working with that data, return data.  There are schematized protocols like OData where you just point the type provider at  the service here.  These angle brackets are saying give me the types and information catalog coming from  the OData Netflix service.  OData is a protocol supported by many Microsoft products, so you can just point it at the  end point for SharePoint OData services or Excel or other products, for example.  There are samples around for doing this with Hadoop and Hive.  This lets you just point at the head node here of a Hadoop cluster.  You give it the port to access on and you access through to the metadata associated with the  tables, the distributed file system tables stored in the Hadoop system.  Again, it just makes working with Hadoop substantially, with HDFS and Hadoop substantially easier.  World Bank, we've seen this one.  You can integrate whole data markets.  You could imagine doing this to all of the Bloomberg or the Reuters catalog of information,  for example, and integrating all of that into a strongly type setting.  Old-fashioned protocols like Wizzle, WMI data on the local machine is another massive piece  of schematized information that could be integrated in.  Every base we've seen.  So another really, really interesting use of this mechanism is to access from F-sharp  into another programming language.  So TypeScript is a strong typing or sort of pseudo typing over the top of JavaScript.  And part of what TypeScript can consume are these.ts files, which give strong typings  over existing JavaScript libraries, kind of like header files for JavaScript libraries.  And you can, when you're compiling F-sharp down to JavaScript, using, in this case,  using something called the FunScript sort of dynamic, the FunScript compiler for F-sharp  code, then it's useful to be able to just access all of those TypeScript, all of those  JavaScript libraries through a strong typing. And so you do that with the TypeScript type  provider in F-sharp.  You're in F-sharp here and you point it at a.ts file.  And then you, this is for jQuery and you can do things like j, we've got the j here,  so you can do things like j.jQuery over here.  If you're interested in looking at that, there's, so let's take a look at FunScript  here. This is the home of the FunScript project.  It's again a fully open source project part of the F-sharp open source community. You  can see some great examples of how to use F-sharp to write fully JavaScript HTML5 apps.  And you're using Type providers and Type providers through to the World Bank and movie databases  and all sorts of other things purely from F-sharp code and sitting on top of TypeScript  typings. You can see the various TypeScript typings that are sitting on top of here, Google  Maps or jQuery and Node and so on.  Okay. So FunScript is a great thing to get involved with. There's another F-sharp to  JavaScript system, a professional system called Web-sharp. It's also open source. And that  also I believe has a TypeScript type provider as part of it. And that's a fabulous thing  to get involved with as well. That's this Web-sharp system here.  Okay. All right. So the last thing we'll look at is integrating another programming language  which is R, the R statistical programming environment. And I mean, F-sharp has some nice  visualization libraries with F-sharp.charting. But R has lots and lots of statistical functionality  and visualization functionality which we'd like to be able to access from F-sharp. And  so we can do that with the R-type provider. So let's take a look at what that looks like  here.  So that's the R-type provider. Again, an open source component done by Hedge Fund called  Blue Mountain Capital in New York. And when you reference the R-type provider, then any  R packages you've got installed on your machine become available under this R-provider.namespace  environment. Okay. So all of these different packages which are the base set of packages  that come with R are now available for use from the F-sharp language. Actually, you get  almost better tooling than you get in R in some ways. There's a little bit of friction  at the boundary because R is basically an untyped system, a dynamically-typed system.  But there's enough metadata there to allow us to do some kind of interop through to the  system.  So we can, let's do, use this to, I think, let's check that I have, okay, let's do this  again. Okay. So this is downloading a set of stock prices here and then using this R  data frame library here to build a data frame, a cube of the relationship between the labels  and the stock prices. And then using this sort of standard piece of R visualization machinery  pairs to look at the pairwise correlations between the different stock prices. And you  can work out various statistical things about how, for instance, Microsoft and Apple correlate  here.  Okay. Okay. So F-sharp to R. Okay. So a couple of other themes in this work which I find  personally really intriguing. One is this thing about working with these external data  sources at multiple scales of, of, of schematization. Data scripters need to work with lots of different  scales or granularities of schematization. They might need to work with just all the  entities in the database to search all the entities. For instance, show me any entity  which has got anything to do with New York. Okay. That's one model. Another model is to  dig down into sort of the organizing things by type. So just show me all the different  models of, of, of cars, for example, the table of models of cars. It's also quite interesting  to go dig down to a level to give each individual its own unique type or own unique name. Okay.  And the, we can see that, for example. Let's take a look at what that looks like in free  base. First, have a think about like if you've got six objects, right, then there's, and  there are three of them have type A and three of them have type B. Then there's three different  ways you can present the API. You could just have a, a list of objects. You could have  organized by A and B. Or you could name each individual A, A1, A2, A3, and AB1, B2, B3.  So now when you want to access, let me just load this up here. Okay. Load up free base  again. Okay. Now, if you want to, let's have one more. This one here. Okay. If I want to  find hydrogen, I can do it by name. Or I can actually come along to the individuals property  in the chemical elements here. Chemical elements start individuals. And now I've got a strong  name through to each of the individual entities in this database. And remember there are like  six, there are like 50 million something entities in the database altogether. And each of those  can be given a strong name. By default, we actually just take the first 1000 from each  of the, each of the tables here. So you're just getting the first 1000 individuals or  there's, there's, that come through from the system. And there's less than 1000 chemical  elements. So that's all of the chemical elements. So we can search through those, for example.  Okay. So you can do this with just about everything, with every table automatically. So if you're  interested in a type of getting the, of accessing through to particular infectious diseases,  you can do that here. If you're interested in accessing individual computer scientists,  you can do that here and work with those. Don't tell the NSA about this feature. Okay.  Really, really no one. They already know. Sorry. Okay. Okay. You can see that, you can  see the interesting thing about this actually is in the context of programming with real  world information is these things don't actually disappear. You know, these things are around  for forever. So if we come along to arts and entertainment and we're interested in opera,  for example, and we're interested in operas here, we come along to individuals. It's the  first time I've looked at this one. So just takes a moment to download the set on the  first time. Okay. Then you can see the set of operas coming through here. Okay. Okay.  So that intrigues me because what you see people, as people are working with these information  sets, they start off trying to generalize an algorithm and then something goes wrong  and then they need some sample data to work with them. They need to dig into one or two  individuals in particular. Then they come back and they work at the more general level  again. And so you need, you can give a unique type to every individual in the database if  you want. And so you need a language that can scale to support massive amounts of scalable  metadata to operate at all the different levels. Okay. The next theme I'll bring out  in the last couple of minutes is about units of measure. These information services often  come with units of measure. For instance, the highest wins is in kilometers per hour.  As that information gets transformed into F-sharp, it actually does a normalization  into the SI unit system because F-sharp likes the SI units. So it actually changes it into  meters per second here. So you actually get that metadata being propagated into the type  system of the units of F-sharp so that you know that this top win rating of the Hurricane  Sandy here is actually in meters per second. Okay. A word about schema change. Everyone  always asks about this. What happens when types change on the outside? The more granular  you are, the more fine-grained you are in your projection of metadata, then the more  you're exposed to schema change. But many data sources are really surprisingly stable.  In some ways, I think these data sources are sometimes more stable than our programming  frameworks. You get more churn at conferences like this of new programming frameworks and  new ways of doing things, then you get change in some of these data sources. Some data sources  provide these snapshot dates that let you snapshot either the metadata or the actual  data at a particular point in time, sort of versioning systems, and you can project those  across into the versioning mechanisms of the programming language. If change happens while  you're working with a data source, the provider can give an invalidation signal while you're  working with it. Just to mention that if you're erasing types, so all these types disappear  at runtime through erasure, and that makes your compile code much less fragile to change.  Okay. If you want to use some of these type providers, I've mentioned fsharp.data. It's  an open source library. You can get it on Nougat. Just go to fsharpgithub.io, fsharpdata,  and you can just use it in Visual Studio or on the Mac or Linux or any system that supports  fsharp code. Okay. So just to finish off, the big themes are we live in an information  revolution. There are many aspects to that, but that needs to have ramifications for how  we think about programming languages, how we design our languages, and how we think  about information access. We need to bring information into the languages at internet  scale, and I think we've broken some new ground in how this can be done in practical settings.  If you'd like to find out more, go to fsharp.org. You can try it online in your browsers right  now on trifsharp.org, and there's great documentation on the MSDN Visual Fsharp pages as well. And  thank you very much, and I think we've got a few minutes for questions as well.  Thanks. Questions? Yeah. Yes. Yes. So a part of every type provider is to provide a query  implementation, an Iquirable link implementation effectively here. So when we're accessing the  books database, it's got three million books in it, so we can't possibly bring that table  to the client side. So this is what an fsharp, this part here is what an fsharp link query  looks like. Just says the obvious thing. We're using a freebase operator called approximate  matching, which is a bit looser than exact matching in.NET, and we're just going to  take the top 10 books, and then we can impact that into the show grid function, and that  gives us the books here on the right. Okay. So querying is absolutely essential to nearly  all of these remote information services. Yeah. Yeah.  So we haven't done anything with that particular setting. It's going to work just fine. Your  metadata comes across, you can project that into types, and there's no performance overhead  to any of the metadata integration that we do, because that's a compile time only feature.  The actual runtime behavior is just like it normally would be in any way of accessing  through to the system. Okay. So there's zero compile time overhead to this mechanism unless  you've made a mistake in your code and how you're doing things. Yeah.  Any more questions? Okay. Then I think we're done. Thanks very much.
Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_0_107.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_108_2135.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_2136_5459.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_5460_6179.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_6180_6551.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_6552_13343.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_13344_15347.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_15348_15407.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_15408_15779.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_15780_16535.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_16536_17927.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_17928_18035.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_18036_18575.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_18576_18779.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_18780_18863.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_18864_18959.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_18960_19271.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_19272_19415.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_19416_20363.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_20364_20555.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_20556_20807.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_20808_23147.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_23148_23195.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_23196_24707.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_24708_26171.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_26172_27011.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_27012_27071.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_27072_27239.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_27240_27299.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_27300_28607.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_28608_28667.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_28668_30083.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_30084_31427.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_31428_31487.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_31488_31547.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_31548_31655.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_31656_36467.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_36468_46127.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_46128_46259.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_46260_46739.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_46740_52991.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_52992_53387.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_53388_53663.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_53664_54383.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_54384_56411.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_56412_57023.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_57024_59843.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_59844_61367.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_61368_63035.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_63036_66215.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_66216_66935.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_66936_67055.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_67056_67211.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_67212_68003.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_68004_70019.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_70020_71831.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_71832_71879.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_71880_81263.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_81264_81635.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_81636_81827.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_81828_82415.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_82416_83411.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_83412_83879.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_83880_84851.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_84852_84899.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_84900_85379.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_85380_88931.png Don_Syme_-_Making_Magic_-_Combining_Data__Information__Services_and_Programming__at_Internet-Scale-6_88932_89026.png
Hi everyone. Great to be back in Oslo. It's, I came here for a Java conference last time and it's NDC this time. So my name is Don Sein. I'm a principal researcher at Microsoft Research in Cambridge. I, my mate, I guess some people know me as having worked on C-Sharp and the design of the F-Sharp language. I, F-Sharp, I, as one of the things I'll be bringing out about the F-Sharp programming language today is that one of the exciting things that's going on is it's a strong open source community and I see myself as a contributor to that community rather than as a sort of, sort of leader of that community. And just one of the, one of the many people contributing. And so I prefer to introduce myself these days as a F-Sharp contributor who happens to also be a principal researcher at Microsoft Research. So today I'm going to be talking about what for me has been perhaps the most exciting, the most inspiring, the most intriguing programming language feature I've had the pleasure to work on during my time at Microsoft Research. And it's all about the fundamental problem of how we bring data, information, services and programming together and do this at a scale that's suitable for the modern era. And you can use it today in product form in Visual Studio 2012 in F-Sharp 3.0. You can use it through the open source editions of F-Sharp on all sorts of platforms to access all sorts of information services in many, many different kinds of programming settings. But beyond that, it's actual existence in F-Sharp today. I hope it also serves as an inspiration for what programming can be in the future for a whole range of different languages. We've seen this work have, I know through personal contacts and emails, having direct and strong influence on the designers of C-Sharp and the designers of OCaml and of Scala. And I believe through that and through, as it flows through the industry, we'll see this gradually roll out in many different forms and perhaps many of which we might forget the some of the background that comes from here. But that's sort of the role of research is to come up with ideas which spread of their own accord through many different settings in the industry. So I'm excited from both from a research perspective and also extremely happy with what we've made available to you as developers in F-Sharp 3.0 both in Visual Studio and in the open source editions of F-Sharp. So the fundamental problem as we all know is the world is information rich. It's actually a very, very interesting week to give this line about information richness. And it's, I mean, we know the many, many positive sides of information richness and we know some of the negatives that that might bring and some of the challenges that brings to our society. The way I like to put it is, I mean, as we've known for a long time, the web changes absolutely everything. The web changes the whole way we work as individuals. It enables all sorts of experiences that with using devices that we never thought would be possible. And it also, it changes politics, changes communications, changes many, many things. But this talk is really about how it can go down to the very base level and change some assumptions we make about programming languages and about how we think about the role of programming languages in the world, especially strongly typed programming languages. And the context is the information revolution. This is just one of many ways of visualizing this. This is taken from programmableweb.com about the rise of open APIs on the web. You can see the numbers below here. There's basically an exponential rise in the sort of digital API context that's available on the open web. And we don't expect this kind of thing to stop anytime soon. The size, if you like, the sort of, you know, the magnitude of the information context around us will just continue to grow and grow. And this is the world we have to interact with as programmers. So modern programming is intensely information rich. You see manifestations of that in data science. You see it in programming for devices in that these are now connected always, nearly always connected through to some kind of set of information sources. You see it in the enterprise for a long time. The enterprise has sort of been a microcosm of what is now happening sort of universally across programming. But our languages are information sparse. They don't in themselves contain any information. They don't actually contain for our strongly typed languages. They don't contain representations of information coming from the outside. You won't in a C-sharp library space, you won't find a definition of a chemical element or an amino acid or a football player or any or genome or any of the information sources that people actually have to deal with every day. So instead, our languages contain these very generic information adapters like XML or text or bytes or JSON sort of APIs to help us do sort of untyped generic manipulation of data. And that's great for many purposes. But for many purposes, we need to program against precise data structures. And those need to come from somewhere and be brought into the language from somewhere. So the developer's perspective is in some sense the languages do not integrate information well. If they do, it's sort of either non-intuitive or it's not simple or disorganized or static or high friction. We have to use these generic adapters like web requests or XML readers or JSON. And this is a problem. It's a problem especially for strongly typed languages. Strongly typed languages are incredibly efficient information manipulation devices. Once you have data under the type system of a modern strongly typed language, especially a strongly typed functional programming language, or one with the sort of expressive power of say the.NET and C-Sharp type system, then that's great. Everything kind of goes pretty well. There's still problems. Nulls, for instance, can be a problem in some type systems. But once you have it under the type system, things are good. But if the cost of getting information under the discipline of the type system is too high, then any efficiency you gain by getting information under types is just lost around the edges. And this is one of the main reasons for the big resurgence of interest in dynamically typed programming languages is because you need to interoperate with the outside world more than you need to process things internally in the language. So we need to bring information into the language, integrated into the language at internet scale, strongly tooled and strongly typed. And that's what we're trying to do here. But back in the good old days, the world was neatly separated into statically typed and dynamically typed languages. And everything was simple and the textbooks all were nice and clear. And this has sort of changed a lot because of this problem of integrating with the external world and also the rise of a lot of different kinds of programming as well. So we now see a lot more sort of overlap. And I guess there's a search on in the middle to make statically typed languages more dynamic. You can see that same C sharp 4.0 to make dynamically typed languages more static through, for instance, lots of type inference in JavaScript implementations or to put sort of static facades over dynamically typed languages like in TypeScript. And then there's sort of moderate static typing, some of the guarantees that you get from static typing in limited ways, but expand the use of static typing. And that's what we're doing in this talk here. But before we get into that, I need to tell you about the context that we're doing it in, which is F sharp. And let me, so I'll just run through a little bit of material about F sharp. I'll do this fairly quickly, but just to give you a locator for where we are with F sharp. So these days, I like to separate between two things. One is F sharp as a language and one, the other is a visual F sharp tool. So the visual F sharp tools are part of Visual Studio. F sharp as a language is still designed primarily at, through, at Microsoft and the compiler implementation is done at Microsoft. But F sharp as a language is open source and is cross platform. And this has been a really, I mean, very exciting developments in the F sharp community through the development, through the community of this F sharp software foundation. And this is a software foundation in many ways set up to be orthogonal, to give the reality to F sharp being open source. And F sharp, they see it as their mission to sort of complement what's done by the core language and compiler team at Microsoft and to bring F sharp onto Mac, Linux, Android, iPhone, HTML5 and, and platforms like FreeBSD and even GPGPU programming. And to give sort of a non, I think a non-Microsoft perspective on, on F sharp. I'll talk a bit more about that in a bit. But just to get across that, I think it's worth separating out those two ideas. Visual F sharp tools are part of Visual Studio. F sharp as a language is open source and cross platform. So our mission statement sort of F sharp is strongly typed data immersive functional first language for simple code. We used to say simple code for complex problems. Okay. So actually it's just about simple code when you really kind of come down to it. I mean, you happen to be able to solve a range of complex problems with it. I'll be talking, I'll be giving another talk here today at, just after lunch, talking about functional first programming more broadly and how, how you can succeed with functional first programming in, in industry. And I'll dig into this a bit more of what sort of roles it can play. But code simplicity is absolutely at the core of a lot of what we do in strongly typed functional languages and in F sharp in particular. So there's a function to swap two elements of a tuple and in C sharp that's okay. You can write this in C sharp, but it just kind of gets a bit longer and you take the rotations of a tuple XYZ and you get three, three rotations and yeah, it's fine. But you know, what do you do in C sharp? It's like, it's actually annoyingly hard to write out this kind of simple data manipulation lists and tuples. And it's just one example of many, many examples of F sharp coding which drop out as sort of more simple, more declarative, better properties in terms of robustness and particularly because of the lack of nulls in, in, in F sharp coding. And there's another one reducing FXYZ. You can do the same sort of thing in C sharp. But again, it's just a bit, bit longer and not as declarative. Simple code defining a type, a class in F sharp of one line, it might go on for many lines in C sharp. And you can see that in our data access work which will enable us later on in this talk to write things like this. Get me the, get me the population age of zero to 14 is a percentage of total of Brazil. You know, it kind of doesn't get more simple than that. We're integrating this information from the World Bank Information Service. And it's just about this, it's sort of bringing out that what we're doing in this talk is following on with the general theme. The general mission statement of F sharp about being about simple code carries through to simplicity in data access as well. One thing I'll, I, I, I like to, to, to bring up at this point is why you might not want to use a functional programming language. And this is actually a very humorous piece of satire about, about why you, why you might not want to use a strongly type functional language. Like if you get paid by the line, for example, that's a, you know, then you, you might prefer to write this sort of code. I compare it, you know, with a couple of lines of F sharp code and so on. Love me some curly braces. I can't do without them. Okay. So if you're that kind of program, if you really just can't do without your curly braces, then there, you know, it might not be the right setting for you, but you know, you probably, I reckon you probably can live without those, those curly braces and you can work on that. Okay. How does F sharp get used in practice? I'll, I'll be talking more about this in my other talks. So just a couple of quick examples. One is, for example, application to balance the national power generation schedule for a power company in the UK. It's very common that F sharp gets used as a, as a component inside a larger dot net or maybe a dot net service with a JavaScript HTML5 front end or just within a larger dot net application. One of the interesting things about F sharp is that, is that property that because it's so easy to interrupt not only from F sharp to the rest, for F sharp to access dot net libraries and data services as we'll see in this talk, but you can interoperate from dot net to F sharp very easily, which means you can encapsulate your use of F sharp into however many components you happen to be sort of right for you to use. And that tends to start off as one component and then people expand out their use of F sharp. So Kaggle, for example, Kaggle is a data science company. They run competitions for a sort of crowdsourcing of data analysis. You can enter these competitions and try and do the better data and statistical analysis than other people over these data sets. Kaggle themselves who run these competitions use F sharp for their core data analysis algorithms. And then they, as I said, you start off with some core data analysis algorithms and then you move more and more of your application into F sharp. And the usual things that people report, and this has been a long standing thing with strongly type functional programming, is fewer bugs, easy to refactor, easy to read, and stronger, and because the strong typing gives the better robustness and fewer bugs. Okay. So as I said, F sharp is changing a bit. There was this view, I think, before the F sharp was for Windows. That's just not the case anymore. F sharp runs on many platforms. There's also, I think, a view of Microsoft make F sharp, and that's actually changed a lot. F sharp has many contributors now. The core compiler language is still done by Microsoft, the team at Microsoft, but for Windows and that gets taken and the other contributors take it to other platforms. I think there was a, when we came out with F sharp, when Microsoft come out with something, we have a bit of an overly strong, a bit heavy, I think. So we came out with F sharp from Microsoft's perspective, and that's actually changing to be many different perspectives on what the programming language is for through F sharp.org. And yeah, I guess it's sort of a, it's instead of seeing F sharp as being something that kind of comes from Microsoft for Microsoft's purposes, there's now many different contributors, some of which are Microsoft, some of which are Microsoft research, and in fact Microsoft research have somewhat different goals than Microsoft in contributing to F sharp. Our role is to partly to take programming forward through a language like F sharp. We have everything flowing in from Nougat into the F sharp environment. You can access all the Nougat packages, 11,000 of them from F sharp. There's lots of good community sites. We have Xamarin is a company which provides F sharp support on the iOS and Android platforms along with them primarily C sharp support, but an F sharp option as well, and quite a few startups contributing to F sharp as well. Just to give an example of the exo system around F sharp, that is, not these aren't intrinsically F sharp packages, these are.NET packages, and one of the things about F sharp is it kind of lives in this exo system, is embedded in this exo system of.NET. If you want to access Amazon web service,.NET SDKs, you can do that, and you can access all the great services from Amazon. If you want to access the Azure.NET SDKs, you can use those from F sharp, and they're all open source as well, and you can go and do your cloud computing with either of these platforms. You want to learn more about particular areas of F sharp, data science, machine learning, cloud or data access and so on. Go to F sharp.org, there's lots of good information. You can contribute to those pages. You just send a pull request on the GitHub source for these pages, and then the community organization will integrate your information into that setting as well, so you can get involved with the community foundation. Okay, so back to the main topic, bringing information into the language, at internet scale, strongly tooled and strongly typed. Okay. Right, so let's start with a small piece of information, which is freebase.com. So freebase is an entity graph. You can more or less assume this is taking all of Wikipedia and doing entity extraction, looking at those structured panels of information that are down the right, for instance, that are given organized view over, say, all the different individual entities, individual pages in Wikipedia and actually elsewhere as well, and integrating those into a single web database slash ontology slash service, whatever you want to call it. It's a pre-base, has been acquired by Google and it's a predecessor to what's called the Google Knowledge Graph, which is sort of an even bigger version of freebase. Freebase has got a publicly accessible API, which is what we'll be using. The Google Knowledge Graph, as far as I know, doesn't have that, but it is much bigger. But it's essentially, again, everything that Google sees on the open web run through a set of entity extraction algorithms and graph database wizardry and a whole lot of other amazing stuff that I don't know much about. The end result is an extraordinary sort of source of organized, schematized information about the real world. For instance, you want three million books that you can query and search, then this is a good place to find them. If you want whole databases of TV and film and music and sports and government and just everything, you can find thousands and millions of topics under here. A lot of people, a lot of different companies are getting into entity extraction. This information is useful for all sorts of things. One of the things it's mainly used for is search to provide those lovely little structured panels you get either from Bing or from Google when you type in something into the search engines. They kind of use it to give a more entity-based structured view of the world around us. But wouldn't it be nice if we could just access all of this information from the programming perspective and do it in a sort of strongly typed and organized way? So we might, for example, be trying to do something like, where's my talk gone? Okay, now I've got to skip through. We might be after something like accessing the chemical elements as part of a chemistry app for a phone or for a web or for an educational application. And we'd like to access some basic information about chemistry. Okay, before I begin, I'll just do a little bit of F-sharp programming here. What I'm using here is F-sharp interactive in Visual Studio. I'll just put this down the bottom here. I've created myself an F-sharp script and I've loaded up some scripts up above. The most important one of which here is this one which is called show, which provides me with a function called show, which allows me to take some data. I've created here the numbers from 0 to 99 and I can call the show function on it and display the data over in the right-hand window. I can also do the same thing by piping, using this piping syntax, if I change that to 59 and then show the numbers over on the right. Now, I'm taking the numbers and I'm piping them into the show function here. You'll see me use that quite a lot. I can also use this show grid function to show a more sort of tabular display. And for instance, if I get the table of squares and I do a show grid over on the right, you'll see the values coming out. For some reason, it's showing duplicates when I'm not sure why that is. That's a bug in the show function, in the show grid. I'll also do some charting, so we'll take that data of the table of squares. So this is the numbers for i from 0 to 99 generating i and the square of i. And that's our data there, that's our data set. And we can pass that data set to the charting library, which will then just display the data set in the obvious way. Get rid of this. So let's take a random point cloud, for example, so 10,000 points where we're multiplying two random numbers here, across here, and you see them drift down to an average of about a quarter down here. Okay, just to show how we might do interactive data work with F-sharp, in this one we're going to take a stream of events coming from this window over here. And as the stream comes in, we're going to collect up a data set and display that data set in the chart over on the left, so that as I move my mouse over on the right, we're accumulating a data set from the events on the outside and displaying the data set over on the left. Okay, so that was to take the mouse move event to extract out the x and the y coordinates from the mouse move and to create an updating chart from updating chart.line. This is using, if you're wondering, this is using the F-sharp dot charting library, which is, and in the latest edition of F-sharp charting, you'll see that updating chart has actually been renamed to live chart if you're going to try and do this at home. Okay, so to take another one here, we're going to take, again, showing how functional programming plays with working with data and information. We're going to take the mouse move events from over here. We're going to extract the y coordinates. We're going to sample them every 30 milliseconds. We're going to take a 3,000 millisecond, three-second interview, interval over that, and collect up that data set, updating data set, and we'll have that displayed over here, and you can see our data set has been generated. And I'd like you to think, how long would it take you to do that kind of data specification and transformation and visualization in an object-oriented kind of way? Okay. So this is fundamentally, you want combinators like sampling, you want them like taking a window, that's exactly what you think about when specifying the problem is, I want to extract out the y coordinates, I want to sample, I want a window time interval, and I want an updating chart of that data set. Right, but let's look at information integration. Okay, so what we've done with F-sharp 3.0 is really, part of the magic of it is that it's just such a simple change to the language, okay, which is that instead of referencing a library, we also use to the idea of referencing a DLL, okay, we know that in.NET, or.NET DLL contains a set of types, a set of classes, a set of namespaces in classes, and we know that in the Java context, you reference a JAR file, it's the same thing, it contains some types, okay. So why not move to a provider model for libraries instead of a fixed set of types? So instead of referencing just a finite collection of types, we're actually referencing a component which computes types on, types and the contents of types on demand, okay. So we'll go through what that means in more detail in a bit, but effectively it's just doing what we often do in computing, which is when we have something, we turn to use a provider model for that something instead. So when I'm referencing Freebase here, that Freebase.dll only contains one or two types, okay, but those are special components which can compute types on demand, okay, and compute them at compile time and compute them in the development environment. This DLL acts as a plug-in to Visual Studio and it also acts as a plug-in to my execution environment down below. So we're moving to a provider model for types and that DLL here lets us have this Freebase data provider and we give it some static parameters. We tell it how we're going to access the service, so give it the API key to our service, which is known at compile time to access the metadata for Freebase and we give it another parameter which I'll explain more in a bit. And once we have that executing the lines through to this data line, then the type of this data object is now represents the entire universe of information that's available through Freebase, okay. So that's effectively handled to a structured type view of everything on Wikipedia. So that means when we start to navigate the properties of this data object, we're actually sort of navigating not at runtime where we were actually accessing the data, but at compile time where navigating the metadata associated with this service. Okay, so in many cases I haven't actually previously accessed these, I've never looked at that engineering domain before, okay. So you're interested in knowing things about engineering and you're knowing, for instance, what different kinds of batteries are available in the world, then we can at compile time get strong typing and strong tooling to access all the different parts of this information service. So for instance astronomy, we can come through, we can come through to astronomers or to stars here. One moment. Okay, to stars or star systems and all the different tables of information available in the information service about astronomy. So let's take a look at some of the actual data, for instance, we can come through to biology and we can look at the amino acids. The double ticks used in F-sharp are just because the name contains a space which is very common in this kind of working. So let's get the table of amino acids and we have it available over on the right here. Okay, so now this is strongly typed data, although it might look like it is a sort of a dynamic language feature, so if I actually iterate the amino acids here and I look and I try and do completion, we haven't actually iterated these, I'm trying to do completion on this, that we get a strong type for what the amino acid A actually is and that means we can get strong completion on all the properties of the amino acid known to the information service. And what strong typing gives us is not just the usual properties of strong typing which is like code correctness, for example if I make a mistake here about misspelling amino acids up here then I'll get an error telling me that table doesn't exist and that's all the good things that come from strong typing, but it also gives us strong tooling in that when as we come along and we look at something like the, well there's a good one for instance, so we look at the CAS ID here, we actually see this field is deprecated, we get strong information coming through our compile time telling us that the information service is changing in some way and that we might need to go and use some other property. Okay, so all the usual benefits of strong typing, but in this context where we're working with that sea of information on the outside, as long as schema information can be flowed into the programming language we can make use of it. Okay, right. Okay, so what we've done there, we've integrated all of freebase.com through using a simple change to the programming language which is to use a provider model for libraries, an on-demand compile time computed metadata projection from the information services into the type system of the host programming language. Okay, so let's take a look at a second example and then I'll dig into more about how this actually works in practice. Okay, this one is going to be looking at the World Bank here. So the World Bank is very typical of the information revolution and organization whose mission is primarily about helping the developing world through finance and funding also becomes a data organization. And one part of their mission is to collect and provide data about, data time series information sets about various indicators about countries and regions about the world. So we can, on their website, we can go down and dig in to look for instance at Australia and you can find out all about, all of a range of different development indicators about each of these different countries. You can see the huge number of indicators that are available for access. And what we'd like is all of this to be sort of automatically projected into the programming language for us to work with. And that's what this provider does here. It takes all of that metadata and makes it available as F-sharp, as to the F-sharp system, the F-sharp compiler development environment tools in a strongly type form. And this provider, the source code is available and I'll give this reference again. This is actually available now in the F-sharp.data library. It's open source library on GitHub. I'll give a reference to that later. Some of the namespaces will have changed. This is actually part of the F-sharp.data.dll library now. You can go and look at the source code for this provider. It's only about 500 lines of code or something to do the projection from that information service into the, into F-sharp. And it's a lovely little sample of a standalone type provider. Okay, so once we have the reference to the library, we can get a data context through to the World Bank. And now we have all of that information in the World Bank available. We can go through to individual countries like we're doing here. We can choose Greece, for example. And one of the things we get is the indicator set that's available for Greece. And all of those 4,000 or so different indicators coming from the World Bank as time series data sets are now available for us to use directly in the programming language. So we're interested, for instance, in debt in Greece. I can't imagine why we'd be interested, but maybe we would. You can see there's actually just an incredible number of different kinds of debt in the world. It's just really, it's just, it goes on and on and on. So, but whichever ones you're interested, you can find the time series data sets here. Once you get the, it's a time series here, so it has, you can get either the values associated that or the dates and times associated with the entries in the data set. So let's take a look at some of the data now. Okay, for instance, we'll get a set of countries here. And we'll do a combined chart for the population statistics for those countries. Okay, so you can see how the populations are growing over time for the, with China and India at the top. Okay. Okay, so we could easily just come along and choose a different population statistics. You might be interested in how the youth population is growing as a percentage of the total. We substitute that in there and combine those here. Okay. It's running along and getting the different indicators and you'll see that how that's changing over time as a percentage. And actually the youth percentage is interestingly changing going down in almost all of those countries. Okay. Okay. And a different view of the same thing as a pie chart for the same population data. Okay. Right. Okay. So integrating, so we've seen two examples, integrating all of freebase.com as if it were a library and integrating all of the World Bank's data as if it were a library. Type provider feels just like a library to the programmer. It's a design time component that computes a space of types and methods on demand. It's an adapt, you can think of it as an adapter between the metadata on the outside and the.NET type system on the inside. You can, from a technical point of view, think of it as sort of on demand, scalable, compile time provision of type module definitions. It's a bit, it's a form of compile time metaprogramming but a very simple and sort of straightforward. It's a lot simpler than adding macros to your language. And languages that add macros usually don't do it in an on demand way where the macros are computed lazily, effectively. Haskell, for example, has a macro system but that's, it's funny, it's a lazy language with a strict macro system. F-sharp is a strict language with a lazy macro system. Okay. So on demand types makes your language internet scalable for metadata integration. So how does this work? When we do database, when we, the first line, freebase.getdata context, the compiler or the IDE, it's effectively getting your symbols from the web, from the providers. It asks the provider for about the symbol get data context and the provider says, okay, it's got a return type of freebase data context, whatever that is. Then the compiler sees the code data. Then the compiler IDE will request the contents of the type freebase data context. Well the provider, the component then has to go and ask the freebase metadata service for what's the contents of this type. And then the provider reports the top level domains for freebase and one of them is society, for example. The compiler sees this code and it asks for more information about society and on and on it goes. Okay. So the symbols and the specifications are being provided on demand and everything is on demand right down to the documentation. You don't have to download, you're only accessing the parts of the metadata information service that you actually need to compile your program, which is what the magic that makes it work. You're not downloading all of the freebase metadata just to compile a three line program. Okay. The next theme that comes out of this is that one data source and one mechanism lets you, it's just got an astonishing number of applications. And the way we like to say this in the F-sharp communities, like all your type are belong to us, that any notion of type that you have in the outside world where you can actually both access it at compile time and access the corresponding data at run time, that can be projected into the F-sharp programming language, which makes a whole world sort of part of the F-sharp exosystem now. So SQL is an obvious example, good old SQL programming. The F-sharp, the Microsoft.fsharp, the data that type providers contains this SQL data connection type provider where you just give a connection string and you get link style programming over the contents of the database straightaway. You can, for instance, use this to mash up several different databases. You can take, it's fantastic for data mashup work because your one script can access many different data sources. Instead of in Visual Studio having to go right click, add data reference, add service reference, build a project and kind of have a big lot of files and generated code, all part of your environment, you've just got a 20 line or 50 line script integrating and incorporating all the different data sources. So one, so more routine example is CSV file data. And you can look at examples of this on the TryF-sharp website. You can actually use type providers in the browser through tryf-sharp.org where you can do some F-sharp programming directly from the browser and that includes type providers in F-sharp 3.0. And this is great. You just point it at a CSV file here. Most CSV files come with a header with some metadata and it actually just infers the types from the actual data in the CSV file. It downloads the prefix of the CSV file, looks through the first number of rows, probably defaults to 10 or so and you ask it to infer the types and ignore any errors coming through. And when you get most, since most open government data sets are actually distributed as CSV files these days, then this lets you integrate external data sets and gives you the same strong type in completion over the contents of those data sets as you can see here. So JSON data, this is another interesting way of getting type specifications. And a typical way of using this is you want to access one of those APIs at the start of the talk. But the API documentation just shows you the JSON data that comes back from the API. But you want that in a strongly typed form. And the typical.NET, you know, Java style as you go to then design a lot of classes and use some serialization framework for mapping these in. And it's kind of clunky. And instead, if the F sharp model, you just use a JSON provider and just give it a sample of the data that comes back. And it's usually a sample taken either directly from a documentation or a test call to the service. And that sample is enough to drive a strong typing for the data. So here's the compile time, you can see the angle brackets, the compile time sample that we're giving, which describes the types by sample. And then once we've got that, we get a parser for those types where we can actually use that to access other data here. And I'll just show you an example of doing that with a Twitter feed here. So here we, what have I done here? Okay, I'll move this up. Okay, authenticate us here. I think I need to copy this in. Maybe I've already, I'll do it anyway, 2559. Okay, so there are actually some nice wrappers for the Twitter API in.NET, but not for all of the Twitter APIs. So for instance, this part to get user suggestions for the active user doesn't have a nice wrapper in the standard Twitter API. So we actually can get a strongly typed wrapper for that very, very simply by downloading some sample data here for the data returned from the API. And we then put that sample data into the JSON type provider as a description of the kind of thing we get back from the API. And now when we actually get the suggestions here, and this could, this same code will work for tomorrow's suggestions and any future data returned from that API because the API doesn't change because it's got a version number 1.1 here which will keep it steady. And then now as we iterate the suggestions here, then we can see the data coming back, the name, the size, and the slug here, whatever the slug means associated with the data. So we have a strongly typed view over the data coming back from the API. A slightly more complex piece of data coming back is this trends data. And we can get the same thing here. We have the trends data. We get a strongly typed view over the trends data. And we can now, so the key thing is here is that at these kind of points we get strong typing here. So we can see that the data. And I haven't even looked closely at this data and I'm already navigating it and noticing that it's got locations information, so for trending locations around the world. And I didn't know that until the strong typing sort of guides us through the process of using it. And so the contents of the data as well, you can see, you can tell whether it's promoted data is associated with events and so on the URL. And we can iterate that data here. Okay, so you can see all the various trends right now coming from Turkey, for example, here. Okay, so types from JSON data, it just makes working with JSON so simple in a strongly typed setting and so sort of frictionless, right? There's no need to be thinking about building object models corresponding to this data. You can do the same thing with XML, the same, exactly the same model, you give it a sample and you get parsers and serializers and the like for working with that data, return data. There are schematized protocols like OData where you just point the type provider at the service here. These angle brackets are saying give me the types and information catalog coming from the OData Netflix service. OData is a protocol supported by many Microsoft products, so you can just point it at the end point for SharePoint OData services or Excel or other products, for example. There are samples around for doing this with Hadoop and Hive. This lets you just point at the head node here of a Hadoop cluster. You give it the port to access on and you access through to the metadata associated with the tables, the distributed file system tables stored in the Hadoop system. Again, it just makes working with Hadoop substantially, with HDFS and Hadoop substantially easier. World Bank, we've seen this one. You can integrate whole data markets. You could imagine doing this to all of the Bloomberg or the Reuters catalog of information, for example, and integrating all of that into a strongly type setting. Old-fashioned protocols like Wizzle, WMI data on the local machine is another massive piece of schematized information that could be integrated in. Every base we've seen. So another really, really interesting use of this mechanism is to access from F-sharp into another programming language. So TypeScript is a strong typing or sort of pseudo typing over the top of JavaScript. And part of what TypeScript can consume are these.ts files, which give strong typings over existing JavaScript libraries, kind of like header files for JavaScript libraries. And you can, when you're compiling F-sharp down to JavaScript, using, in this case, using something called the FunScript sort of dynamic, the FunScript compiler for F-sharp code, then it's useful to be able to just access all of those TypeScript, all of those JavaScript libraries through a strong typing. And so you do that with the TypeScript type provider in F-sharp. You're in F-sharp here and you point it at a.ts file. And then you, this is for jQuery and you can do things like j, we've got the j here, so you can do things like j.jQuery over here. If you're interested in looking at that, there's, so let's take a look at FunScript here. This is the home of the FunScript project. It's again a fully open source project part of the F-sharp open source community. You can see some great examples of how to use F-sharp to write fully JavaScript HTML5 apps. And you're using Type providers and Type providers through to the World Bank and movie databases and all sorts of other things purely from F-sharp code and sitting on top of TypeScript typings. You can see the various TypeScript typings that are sitting on top of here, Google Maps or jQuery and Node and so on. Okay. So FunScript is a great thing to get involved with. There's another F-sharp to JavaScript system, a professional system called Web-sharp. It's also open source. And that also I believe has a TypeScript type provider as part of it. And that's a fabulous thing to get involved with as well. That's this Web-sharp system here. Okay. All right. So the last thing we'll look at is integrating another programming language which is R, the R statistical programming environment. And I mean, F-sharp has some nice visualization libraries with F-sharp.charting. But R has lots and lots of statistical functionality and visualization functionality which we'd like to be able to access from F-sharp. And so we can do that with the R-type provider. So let's take a look at what that looks like here. So that's the R-type provider. Again, an open source component done by Hedge Fund called Blue Mountain Capital in New York. And when you reference the R-type provider, then any R packages you've got installed on your machine become available under this R-provider.namespace environment. Okay. So all of these different packages which are the base set of packages that come with R are now available for use from the F-sharp language. Actually, you get almost better tooling than you get in R in some ways. There's a little bit of friction at the boundary because R is basically an untyped system, a dynamically-typed system. But there's enough metadata there to allow us to do some kind of interop through to the system. So we can, let's do, use this to, I think, let's check that I have, okay, let's do this again. Okay. So this is downloading a set of stock prices here and then using this R data frame library here to build a data frame, a cube of the relationship between the labels and the stock prices. And then using this sort of standard piece of R visualization machinery pairs to look at the pairwise correlations between the different stock prices. And you can work out various statistical things about how, for instance, Microsoft and Apple correlate here. Okay. Okay. So F-sharp to R. Okay. So a couple of other themes in this work which I find personally really intriguing. One is this thing about working with these external data sources at multiple scales of, of, of schematization. Data scripters need to work with lots of different scales or granularities of schematization. They might need to work with just all the entities in the database to search all the entities. For instance, show me any entity which has got anything to do with New York. Okay. That's one model. Another model is to dig down into sort of the organizing things by type. So just show me all the different models of, of, of cars, for example, the table of models of cars. It's also quite interesting to go dig down to a level to give each individual its own unique type or own unique name. Okay. And the, we can see that, for example. Let's take a look at what that looks like in free base. First, have a think about like if you've got six objects, right, then there's, and there are three of them have type A and three of them have type B. Then there's three different ways you can present the API. You could just have a, a list of objects. You could have organized by A and B. Or you could name each individual A, A1, A2, A3, and AB1, B2, B3. So now when you want to access, let me just load this up here. Okay. Load up free base again. Okay. Now, if you want to, let's have one more. This one here. Okay. If I want to find hydrogen, I can do it by name. Or I can actually come along to the individuals property in the chemical elements here. Chemical elements start individuals. And now I've got a strong name through to each of the individual entities in this database. And remember there are like six, there are like 50 million something entities in the database altogether. And each of those can be given a strong name. By default, we actually just take the first 1000 from each of the, each of the tables here. So you're just getting the first 1000 individuals or there's, there's, that come through from the system. And there's less than 1000 chemical elements. So that's all of the chemical elements. So we can search through those, for example. Okay. So you can do this with just about everything, with every table automatically. So if you're interested in a type of getting the, of accessing through to particular infectious diseases, you can do that here. If you're interested in accessing individual computer scientists, you can do that here and work with those. Don't tell the NSA about this feature. Okay. Really, really no one. They already know. Sorry. Okay. Okay. You can see that, you can see the interesting thing about this actually is in the context of programming with real world information is these things don't actually disappear. You know, these things are around for forever. So if we come along to arts and entertainment and we're interested in opera, for example, and we're interested in operas here, we come along to individuals. It's the first time I've looked at this one. So just takes a moment to download the set on the first time. Okay. Then you can see the set of operas coming through here. Okay. Okay. So that intrigues me because what you see people, as people are working with these information sets, they start off trying to generalize an algorithm and then something goes wrong and then they need some sample data to work with them. They need to dig into one or two individuals in particular. Then they come back and they work at the more general level again. And so you need, you can give a unique type to every individual in the database if you want. And so you need a language that can scale to support massive amounts of scalable metadata to operate at all the different levels. Okay. The next theme I'll bring out in the last couple of minutes is about units of measure. These information services often come with units of measure. For instance, the highest wins is in kilometers per hour. As that information gets transformed into F-sharp, it actually does a normalization into the SI unit system because F-sharp likes the SI units. So it actually changes it into meters per second here. So you actually get that metadata being propagated into the type system of the units of F-sharp so that you know that this top win rating of the Hurricane Sandy here is actually in meters per second. Okay. A word about schema change. Everyone always asks about this. What happens when types change on the outside? The more granular you are, the more fine-grained you are in your projection of metadata, then the more you're exposed to schema change. But many data sources are really surprisingly stable. In some ways, I think these data sources are sometimes more stable than our programming frameworks. You get more churn at conferences like this of new programming frameworks and new ways of doing things, then you get change in some of these data sources. Some data sources provide these snapshot dates that let you snapshot either the metadata or the actual data at a particular point in time, sort of versioning systems, and you can project those across into the versioning mechanisms of the programming language. If change happens while you're working with a data source, the provider can give an invalidation signal while you're working with it. Just to mention that if you're erasing types, so all these types disappear at runtime through erasure, and that makes your compile code much less fragile to change. Okay. If you want to use some of these type providers, I've mentioned fsharp.data. It's an open source library. You can get it on Nougat. Just go to fsharpgithub.io, fsharpdata, and you can just use it in Visual Studio or on the Mac or Linux or any system that supports fsharp code. Okay. So just to finish off, the big themes are we live in an information revolution. There are many aspects to that, but that needs to have ramifications for how we think about programming languages, how we design our languages, and how we think about information access. We need to bring information into the languages at internet scale, and I think we've broken some new ground in how this can be done in practical settings. If you'd like to find out more, go to fsharp.org. You can try it online in your browsers right now on trifsharp.org, and there's great documentation on the MSDN Visual Fsharp pages as well. And thank you very much, and I think we've got a few minutes for questions as well. Thanks. Questions? Yeah. Yes. Yes. So a part of every type provider is to provide a query implementation, an Iquirable link implementation effectively here. So when we're accessing the books database, it's got three million books in it, so we can't possibly bring that table to the client side. So this is what an fsharp, this part here is what an fsharp link query looks like. Just says the obvious thing. We're using a freebase operator called approximate matching, which is a bit looser than exact matching in.NET, and we're just going to take the top 10 books, and then we can impact that into the show grid function, and that gives us the books here on the right. Okay. So querying is absolutely essential to nearly all of these remote information services. Yeah. Yeah. So we haven't done anything with that particular setting. It's going to work just fine. Your metadata comes across, you can project that into types, and there's no performance overhead to any of the metadata integration that we do, because that's a compile time only feature. The actual runtime behavior is just like it normally would be in any way of accessing through to the system. Okay. So there's zero compile time overhead to this mechanism unless you've made a mistake in your code and how you're doing things. Yeah. Any more questions? Okay. Then I think we're done. Thanks very much.
a scale that's suitable for the modern era. And that might bring and some of the challenges that web changes absolutely everything. The web programming is intensely information rich. You sources. You see it in the enterprise for a long information sparse. They don't in themselves contain for our strongly typed languages. They languages are incredibly efficient information language, especially a strongly typed functional typing. And that's what we're doing in this talk Microsoft and the compiler implementation is done functional programming language. And this is libraries and data services as we'll see in this being something that kind of comes from Microsoft contributing to F sharp as well. Just to give an the Google Knowledge Graph, which is sort of an strong typing, but it also gives us strong applications. And the way we like to say this in so types from JSON data, it just makes working example. Let's take a look at what that looks Then they come back and they work at the more database if you want. And so you need a language how we think about programming languages, how we design our languages, and how we think about