Mining Precision Interfaces From Query Logs
Zhang, Qianrui
Zhang, Haoci
Sellam, Thibault
Wu, Eugene
Interactive tools make data analysis more efficient and more accessible to end-users by hiding the underlying query complexity and exposing interactive widgets for the parts of the query that matter to the analysis. However, creating custom tailored (i.e., precise) interfaces is very costly, and automated approaches are desirable. We propose a syntactic approach that uses queries from an analysis to generate a tailored interface. We model interface widgets as functions I(q) -> q' that modify the current analysis query q, and interfaces as the set of queries that its widgets can express. Our system, Precision Interfaces, analyzes structural changes between input queries from an analysis, and generates an output interface with widgets to express those changes. Our experiments on the Sloan Digital Sky Survey query log suggest that Precision Interfaces can generate useful interfaces for simple unanticipated tasks, and our optimizations can generate interfaces from logs of up to 10,000 queries in <10s.
 This is work with Owen, Houchie, Thibault and myself at Columbia as part of the Data Science Institute. I wanted to start by talking about a specific type of interface, which is filling expenses at Columbia. Basically, we have to fill out this Excel spreadsheet in order to then send it to an admin along with receipts and a bunch of other stuff. Then hopefully we get our money back. No one really likes doing this because it takes a huge amount of time. It's super frustrating. It's kind of very similar if you use other very general purpose apps that happen to be able to do other things that you want. People are really not asking the right question. It's not whether Excel or these systems can do what you want, but whether or not you should. If you think about data analysis, we have a very similar paradigm where for the most part the dominant ways that we interact with the database would be through Excel or some general purpose data exploration application, whereas yes, you can do things that you want to do, but if you have a very specific purpose, then users end up conforming their own use cases and the way they perform analysis to the interfaces rather than the other way around. If you look at the web or mobile, it's kind of the opposite. Instead of Excel spreadsheets, you have to have an expensive one, which does just one thing and one thing very well. It's very easy to use and it's kind of a pleasure. Similarly, instead of opening up Photoshop and using this crazy complicated system, we have very custom filter applications in order to make yourself look more beautiful or the landscape more beautiful. Sometimes if you also have developed specialized applications for interactive visualization, you can also enjoy this kind of simplicity. You want me to use this. Either this or this. One or the other. That is the case. Yes, this is the power of customized interfaces and does it spark joy in people? Yes, Marie Kondo, it does spark joy. Now the question is how do you go about creating these? What are your options? Well, one is of course you can pay people money and hire engineers to figure out what you want and then build an interface and then get something. Of course, money is not raining out of the sky today and so we need some other alternatives. Another thing you might do is to say, well, maybe the end user who knows what they want to do can use a dashboard builder or some other application in order to construct something for themselves. But fundamentally, these interfaces are either very difficult to use or they restrict the kind of analyses you can create in order to keep things simple for end users. Ideally what you would do is you would generate an interface automatically if magic could happen. In fact, in our community, JAG and others have actually done a lot of really interesting work in creating search or crowd type interfaces for databases. It takes a database as input and generates forms such as these for searching or adding new records. Now, they are not analysis specific and so if you have a large database with lots of tables and lots of attributes, it might not be obvious what type of interface to actually create. Now, a very related piece of work is actually visualization recommendation where given data or some previous viz, you want to recommend some other visualization. So the thing that they don't do is actually create the interactive controls which is the focus of our work. Our idea is to why not just go directly with the analysis task in order to generate the interface. That's kind of what we would like to do but it's very hard to know how do you represent and how do you actually get the analysis task. One thing you could do is imagine you could open up someone's brain and try to figure it out from that way but it could be a little bit messy. The approach that we're taking is kind of like a proxy for now is to leverage query logs. These are basically logs of analyses and queries that people have performed in the past already. You can think of them as some representation of at least analyses people have tried to perform. The nice thing is they're pretty much ubiquitous. You could instrument Jupyter notebooks, database recovery logs, provenance is increasingly ubiquitous in everywhere and so on. This already gives us some leverage. The hard work now, I would say the main contribution if you're to skip most of the paper is to define a model to go from a bunch of text that represents the queries on one side into some representation of the set of controls in the interface on the right. I would say that is the primary contribution and then we take a syntactic approach to this. What that means is all we take as input are the queries and the grammar and then we try to generate an interface without looking at the database or having access to database internals. I want to give you an overview of the approach in terms of how we think about this problem. Really our mental model comes from the reverse. If you think about what an interface is, it composes two components. The first is you run a query and then you get the result and you visualise it. Then you have a set of interactive controls that allow you to systematically transform the underlying query in some way. If you imagine transforming it in all possible different ways using these controls, then what that represents is effectively its expressiveness, some set of queries that it can express. What we're doing is the opposite. We're starting with some sample of that and we want to identify essentially the set of widgets that we would need as well as the initial query in order to express these queries. That's what we're going to do. We basically do two steps. One is given these queries, we would like to mine them for interactions and then we would like to map them onto widgets that we can then generate interfaces with. The hard part is going to be generating and identifying these widgets. To start from the left, we model queries as abstract syntax trees. What that means is instead of these string expressions here like average and count, what we're going to do is we're going to parse them into these abstract syntax trees and then look at differences in that space. The nice thing is it then gives us a little bit of semantic information but without having access to the database. This is an abstract syntax tree for a very, very simple SQL query. It can be a little bit complicated. What we would like to do now is mine these interactions. What that means is in this case, there are really two ways that we can go from query one to query two. We can either wholesale swap the entire tree for each other or we can notice that we want to transform average to count and A to B. Both of those are totally valid and both of those are ways that the user could try to express it. What we can think of now is query one and query two constitute two nodes in a graph and then we have different types of colored edges that correspond to the types of transformations in the subtree, in these trees. If we did, so these are correspond to the location in the abstract syntax tree where you're performing the transformation as well as what you're actually transforming. If you do this between different pairs in the query log, then we end up with what we call the interaction graph, where queries correspond to nodes and then these interactions correspond to edges. For example, maybe here we have three queries, Q1, Q2, Q3, where at a particular location we want to swap A to B for the arguments and then maybe later on B to C to go from Q3 to Q2. Now, what we want to do is map this set of edges to a widget that can express these. For example, we might say, well, a dropdown that changes at this particular location and is instantiated with the options A, B, and C might be a useful thing. If the user now clicks on A, then what they're really doing is they're transforming this part of the query to using this particular subtree. Of course, there's many other representations of widgets that you could imagine. For example, you could also use a text box instead. That gives you a little bit more expressive power, but might be a little bit more difficult to use. Clearly, you can see that there's a trade-off that we would like to be able to express as well. What we're going to do as part of this mapping task is we're going to define a library of widgets. The key part is how do you express these widget types? What I mean by that is you can imagine that different widgets have different types of allowable domains. For example, sliders really only make sense if the subtree can be numeric in some nature. Similarly, a dropdown really should be restricted to K subtrees and so on. You can think of these as hard constraints. Whereas, soft constraints now describe these cost functions to say what my preferences might be. It turns out the UI literature and the HCI literature have studied this very, very deeply. We're not going to innovate on this area, except to say that you can encode things like user preference or ease of use and usability and these sorts of metrics into a very simple polynomial cost function. Basically, we do this and we just say we're going to model this as a quadratic equation for each type of widget in order to model how easy it is to use. Given all of these definitions, ultimately what we want to do is given a query log and a library of these widget types, we would like to generate the lowest cost interface as defined by these cost functions that expresses all the queries in the log, meaning that all the queries in the log are reachable in this graph via the widgets that I've selected. It's pretty easy to show that this is a hard problem and we use some heuristics to solve this problem. You can probably come up with pretty good heuristics as well. This hopefully gives you a sense of how we model and think about this problem. What I wanted to do is give you a very short demo to see this in action. This works. This is kind of like Jupyter Notebook except it's this thing called franchise. What we can do in this very, very trivial demo is we can write a query. I can say select these two attributes from an error pollution data set. Here, basically, I can say I want to render as scatter plot. This is what you expect. Now I might say, well, I actually want to see this for particular days. I might want to filter on a day. At this point, if this is the only thing I ever want to do, then there's really only a couple options that I have. Let's say now I say, oh, I want to change the day a little bit. We see that the widgets have changed. You don't have to change widgets like this, but you can. Let's say if I turn it to 30, then we recognize that this is actually the focus point of the analysis, and then now it gives me the slider that I can then now use to vary and stop typing. So this is one use case where as you're performing analysis, you can generate these interfaces for free and then throw them away when you're done with it. But you can imagine other types of use cases as well. All right. Let's see. I want to talk a little bit about experiments. So overall, we ran a bunch of experiments to understand how effective our algorithms are. And to be perfectly frank, it works for arbitrarily complex queries, but the transformations between them need to have some structure that we can exploit. And so that's nice as an alternative to editing and manually typing SQL. So that's nice. So this is kind of an example where this is a very simple query log from SDSS, the Sloan Digital Sky Survey query log, and then this is kind of like the set of widgets that we end up generating. So they seem to be sensible. And of course, it depends. And so if you have ad hoc queries or more complicated transformations, then of course, we're not going to do very well. Now there's kind of like one experiment that I wanted to show you for the sake of time, which is a very short user study that we did. And this kind of is nice because it highlights one kind of like use case where it might be nice to be able to automatically generate interfaces. So here SDSS already has their own web form to perform a bunch of tasks. Essentially of the form, I want to find stars or celestial objects based on a bunch of criteria. And they actually have this document that describes a set of tasks that this web form is well suited for. So what we did was we took a sample of the query log and then we used prison interfaces to generate a set of widgets. Now the layout and everything is manually laid out. And so what you should focus on is just the fact that we've generated a pile of widgets. Now even here, you can notice a couple things. First is there are portions of the original form that we didn't bother to generate. And part of it is because we didn't see evidence of the need to generate those ahead of time. So you could argue it's simplification or that we've kind of lost the ability to do things. Either way, that's kind of what we've done. The other thing is we asked users to perform four of the tasks and to see how well they perform in terms of like time in order to submit the task as well as whether or not it was accurate. For the most part, actually, we don't do better or worse than the original interface except for task one where we did much better. And it turned out the reason was because the original form actually didn't have controls to express this particular task, which is very simple. It's really just looking up objects by their object ID. And turned out that there were other ways, for example, manually typing queries where you can actually do it using the original form, but you end up needing to know the schema and then you know SQL and be able to type things. And so this is kind of like one example where perhaps in many cases you don't actually know upfront when you're actually designing interfaces what the actual analysis you intend to be. And so it'd be nice to dynamically know this after the fact and generate custom interfaces for those. So that's kind of like one possibility how you might use such a system. All right. So I think I have a little bit of time to go into commentary. So basically to come back, the purpose of this work and kind of the sub-area of databases I think about is basically human data interfaces. How do you design interfaces and what is the way that you kind of interact with data? And I would argue that this is actually like fundamentally a very nice data management problem, right? And we have a lot of opportunity to make big contributions in this emerging area called human data interaction. And one way to think about this is in the US we have a couple million software engineers that can do very powerful things and build very powerful tools. But of course that is about 100x smaller than the rest of the population, right? Even if you just look at service workers, there's still 30x people that all want to work with data, right, in banks and insurance companies and governments and so on. And they don't really have this technical expertise. And so the question is what kind of tools can we use to scale out or to kind of like enable existing software engineers or non-technical people to basically scale out and have the disability, right? And so that's something that's worth thinking about. And it is fundamentally a data management problem. It's like how you interact and manage data. And so all I'm saying is outside of efficiency there's many other criteria that are also worthwhile to think about. And so this is a small piece towards what we consider large-scale customized visual analysis interface generation. And our main contribution is this unified model and this way of thinking about it. And of course there's a lot of work to do on our side as well. We don't really think about semantics. We don't think about the database, the data, right? We don't think about generating labels, layout. There's a lot of things we don't do. But this is kind of like the approach that we're going to take and how we're going to move forward. So this is all part of the data visualization management project at Columbia. And so at this point I'm happy to take questions. Thank you. Thanks so much. Goodbye.
 This is work with Owen, Houchie, Thibault and myself at Columbia as part of the Data  Science Institute.  I wanted to start by talking about a specific type of interface, which is filling expenses  at Columbia.  Basically, we have to fill out this Excel spreadsheet in order to then send it to an  admin along with receipts and a bunch of other stuff.  Then hopefully we get our money back.  No one really likes doing this because it takes a huge amount of time.  It's super frustrating.  It's kind of very similar if you use other very general purpose apps that happen to  be able to do other things that you want.  People are really not asking the right question.  It's not whether Excel or these systems can do what you want, but whether or not you should.  If you think about data analysis, we have a very similar paradigm where for the most  part the dominant ways that we interact with the database would be through Excel or some  general purpose data exploration application, whereas yes, you can do things that you want  to do, but if you have a very specific purpose, then users end up conforming their own use  cases and the way they perform analysis to the interfaces rather than the other way around.  If you look at the web or mobile, it's kind of the opposite.  Instead of Excel spreadsheets, you have to have an expensive one, which does just one  thing and one thing very well.  It's very easy to use and it's kind of a pleasure.  Similarly, instead of opening up Photoshop and using this crazy complicated system, we  have very custom filter applications in order to make yourself look more beautiful or the  landscape more beautiful.  Sometimes if you also have developed specialized applications for interactive visualization,  you can also enjoy this kind of simplicity.  You want me to use this.  Either this or this.  One or the other.  That is the case.  Yes, this is the power of customized interfaces and does it spark joy in people?  Yes, Marie Kondo, it does spark joy.  Now the question is how do you go about creating these?  What are your options?  Well, one is of course you can pay people money and hire engineers to figure out what  you want and then build an interface and then get something.  Of course, money is not raining out of the sky today and so we need some other alternatives.  Another thing you might do is to say, well, maybe the end user who knows what they want  to do can use a dashboard builder or some other application in order to construct something  for themselves.  But fundamentally, these interfaces are either very difficult to use or they restrict the  kind of analyses you can create in order to keep things simple for end users.  Ideally what you would do is you would generate an interface automatically if magic could  happen.  In fact, in our community, JAG and others have actually done a lot of really interesting  work in creating search or crowd type interfaces for databases.  It takes a database as input and generates forms such as these for searching or adding  new records.  Now, they are not analysis specific and so if you have a large database with lots of  tables and lots of attributes, it might not be obvious what type of interface to actually  create.  Now, a very related piece of work is actually visualization recommendation where given data  or some previous viz, you want to recommend some other visualization.  So the thing that they don't do is actually create the interactive controls which is the  focus of our work.  Our idea is to why not just go directly with the analysis task in order to generate the  interface.  That's kind of what we would like to do but it's very hard to know how do you represent  and how do you actually get the analysis task.  One thing you could do is imagine you could open up someone's brain and try to figure  it out from that way but it could be a little bit messy.  The approach that we're taking is kind of like a proxy for now is to leverage query  logs.  These are basically logs of analyses and queries that people have performed in the past already.  You can think of them as some representation of at least analyses people have tried to  perform.  The nice thing is they're pretty much ubiquitous.  You could instrument Jupyter notebooks, database recovery logs, provenance is increasingly  ubiquitous in everywhere and so on.  This already gives us some leverage.  The hard work now, I would say the main contribution if you're to skip most of the paper is to  define a model to go from a bunch of text that represents the queries on one side into  some representation of the set of controls in the interface on the right.  I would say that is the primary contribution and then we take a syntactic approach to this.  What that means is all we take as input are the queries and the grammar and then we try  to generate an interface without looking at the database or having access to database  internals.  I want to give you an overview of the approach in terms of how we think about this problem.  Really our mental model comes from the reverse.  If you think about what an interface is, it composes two components.  The first is you run a query and then you get the result and you visualise it.  Then you have a set of interactive controls that allow you to systematically transform  the underlying query in some way.  If you imagine transforming it in all possible different ways using these controls, then what  that represents is effectively its expressiveness, some set of queries that it can express.  What we're doing is the opposite.  We're starting with some sample of that and we want to identify essentially the set of  widgets that we would need as well as the initial query in order to express these queries.  That's what we're going to do.  We basically do two steps.  One is given these queries, we would like to mine them for interactions and then we  would like to map them onto widgets that we can then generate interfaces with.  The hard part is going to be generating and identifying these widgets.  To start from the left, we model queries as abstract syntax trees.  What that means is instead of these string expressions here like average and count, what  we're going to do is we're going to parse them into these abstract syntax trees and  then look at differences in that space.  The nice thing is it then gives us a little bit of semantic information but without having  access to the database.  This is an abstract syntax tree for a very, very simple SQL query.  It can be a little bit complicated.  What we would like to do now is mine these interactions.  What that means is in this case, there are really two ways that we can go from query  one to query two.  We can either wholesale swap the entire tree for each other or we can notice that we want  to transform average to count and A to B. Both of those are totally valid and both of  those are ways that the user could try to express it.  What we can think of now is query one and query two constitute two nodes in a graph and then  we have different types of colored edges that correspond to the types of transformations  in the subtree, in these trees.  If we did, so these are correspond to the location in the abstract syntax tree where  you're performing the transformation as well as what you're actually transforming.  If you do this between different pairs in the query log, then we end up with what we  call the interaction graph, where queries correspond to nodes and then these interactions  correspond to edges.  For example, maybe here we have three queries, Q1, Q2, Q3, where at a particular location  we want to swap A to B for the arguments and then maybe later on B to C to go from Q3 to  Q2.  Now, what we want to do is map this set of edges to a widget that can express these.  For example, we might say, well, a dropdown that changes at this particular location and  is instantiated with the options A, B, and C might be a useful thing.  If the user now clicks on A, then what they're really doing is they're transforming this  part of the query to using this particular subtree.  Of course, there's many other representations of widgets that you could imagine.  For example, you could also use a text box instead.  That gives you a little bit more expressive power, but might be a little bit more difficult  to use.  Clearly, you can see that there's a trade-off that we would like to be able to express as  well.  What we're going to do as part of this mapping task is we're going to define a library of  widgets.  The key part is how do you express these widget types?  What I mean by that is you can imagine that different widgets have different types of  allowable domains.  For example, sliders really only make sense if the subtree can be numeric in some nature.  Similarly, a dropdown really should be restricted to K subtrees and so on.  You can think of these as hard constraints.  Whereas, soft constraints now describe these cost functions to say what my preferences  might be.  It turns out the UI literature and the HCI literature have studied this very, very deeply.  We're not going to innovate on this area, except to say that you can encode things like user  preference or ease of use and usability and these sorts of metrics into a very simple  polynomial cost function.  Basically, we do this and we just say we're going to model this as a quadratic equation  for each type of widget in order to model how easy it is to use.  Given all of these definitions, ultimately what we want to do is given a query log and  a library of these widget types, we would like to generate the lowest cost interface  as defined by these cost functions that expresses all the queries in the log, meaning that  all the queries in the log are reachable in this graph via the widgets that I've selected.  It's pretty easy to show that this is a hard problem and we use some heuristics to solve  this problem.  You can probably come up with pretty good heuristics as well.  This hopefully gives you a sense of how we model and think about this problem.  What I wanted to do is give you a very short demo to see this in action.  This works.  This is kind of like Jupyter Notebook except it's this thing called franchise.  What we can do in this very, very trivial demo is we can write a query.  I can say select these two attributes from an error pollution data set.  Here, basically, I can say I want to render as scatter plot.  This is what you expect.  Now I might say, well, I actually want to see this for particular days.  I might want to filter on a day.  At this point, if this is the only thing I ever want to do, then there's really only  a couple options that I have.  Let's say now I say, oh, I want to change the day a little bit.  We see that the widgets have changed.  You don't have to change widgets like this, but you can.  Let's say if I turn it to 30, then we recognize that this is actually the focus point of the  analysis, and then now it gives me the slider that I can then now use to vary and stop typing.  So this is one use case where as you're performing analysis, you can generate these interfaces  for free and then throw them away when you're done with it.  But you can imagine other types of use cases as well.  All right.  Let's see.  I want to talk a little bit about experiments.  So overall, we ran a bunch of experiments to understand how effective our algorithms are.  And to be perfectly frank, it works for arbitrarily complex queries, but the transformations between  them need to have some structure that we can exploit.  And so that's nice as an alternative to editing and manually typing SQL.  So that's nice.  So this is kind of an example where this is a very simple query log from SDSS, the Sloan  Digital Sky Survey query log, and then this is kind of like the set of widgets that we  end up generating.  So they seem to be sensible.  And of course, it depends.  And so if you have ad hoc queries or more complicated transformations, then of course,  we're not going to do very well.  Now there's kind of like one experiment that I wanted to show you for the sake of time,  which is a very short user study that we did.  And this kind of is nice because it highlights one kind of like use case where it might be  nice to be able to automatically generate interfaces.  So here SDSS already has their own web form to perform a bunch of tasks.  Essentially of the form, I want to find stars or celestial objects based on a bunch of criteria.  And they actually have this document that describes a set of tasks that this web form  is well suited for.  So what we did was we took a sample of the query log and then we used prison interfaces  to generate a set of widgets.  Now the layout and everything is manually laid out.  And so what you should focus on is just the fact that we've generated a pile of widgets.  Now even here, you can notice a couple things.  First is there are portions of the original form that we didn't bother to generate.  And part of it is because we didn't see evidence of the need to generate those ahead of time.  So you could argue it's simplification or that we've kind of lost the ability to do  things.  Either way, that's kind of what we've done.  The other thing is we asked users to perform four of the tasks and to see how well they  perform in terms of like time in order to submit the task as well as whether or not  it was accurate.  For the most part, actually, we don't do better or worse than the original interface  except for task one where we did much better.  And it turned out the reason was because the original form actually didn't have controls  to express this particular task, which is very simple.  It's really just looking up objects by their object ID.  And turned out that there were other ways, for example, manually typing queries where  you can actually do it using the original form, but you end up needing to know the schema  and then you know SQL and be able to type things.  And so this is kind of like one example where perhaps in many cases you don't actually know  upfront when you're actually designing interfaces what the actual analysis you intend to be.  And so it'd be nice to dynamically know this after the fact and generate custom interfaces  for those.  So that's kind of like one possibility how you might use such a system.  All right.  So I think I have a little bit of time to go into commentary.  So basically to come back, the purpose of this work and kind of the sub-area of databases  I think about is basically human data interfaces.  How do you design interfaces and what is the way that you kind of interact with data?  And I would argue that this is actually like fundamentally a very nice data management  problem, right?  And we have a lot of opportunity to make big contributions in this emerging area called  human data interaction.  And one way to think about this is in the US we have a couple million software engineers  that can do very powerful things and build very powerful tools.  But of course that is about 100x smaller than the rest of the population, right?  Even if you just look at service workers, there's still 30x people that all want to  work with data, right, in banks and insurance companies and governments and so on.  And they don't really have this technical expertise.  And so the question is what kind of tools can we use to scale out or to kind of like  enable existing software engineers or non-technical people to basically scale out and have the  disability, right?  And so that's something that's worth thinking about.  And it is fundamentally a data management problem.  It's like how you interact and manage data.  And so all I'm saying is outside of efficiency there's many other criteria that are also  worthwhile to think about.  And so this is a small piece towards what we consider large-scale customized visual  analysis interface generation.  And our main contribution is this unified model and this way of thinking about it.  And of course there's a lot of work to do on our side as well.  We don't really think about semantics.  We don't think about the database, the data, right?  We don't think about generating labels, layout.  There's a lot of things we don't do.  But this is kind of like the approach that we're going to take and how we're going to  move forward.  So this is all part of the data visualization management project at Columbia.  And so at this point I'm happy to take questions.  Thank you.  Thanks so much.  Goodbye.
Zhang_Zhang_Sellam_Wu_0_131.png Zhang_Zhang_Sellam_Wu_132_251.png Zhang_Zhang_Sellam_Wu_252_779.png Zhang_Zhang_Sellam_Wu_780_935.png Zhang_Zhang_Sellam_Wu_936_1235.png Zhang_Zhang_Sellam_Wu_1236_1283.png Zhang_Zhang_Sellam_Wu_1284_14783.png Zhang_Zhang_Sellam_Wu_14784_14879.png Zhang_Zhang_Sellam_Wu_14880_14927.png Zhang_Zhang_Sellam_Wu_14928_15011.png Zhang_Zhang_Sellam_Wu_15012_15947.png Zhang_Zhang_Sellam_Wu_15948_16583.png Zhang_Zhang_Sellam_Wu_16584_16631.png Zhang_Zhang_Sellam_Wu_16632_21251.png Zhang_Zhang_Sellam_Wu_21252_21995.png Zhang_Zhang_Sellam_Wu_21996_22187.png Zhang_Zhang_Sellam_Wu_22188_24107.png Zhang_Zhang_Sellam_Wu_24108_24241.png
This is work with Owen, Houchie, Thibault and myself at Columbia as part of the Data Science Institute. I wanted to start by talking about a specific type of interface, which is filling expenses at Columbia. Basically, we have to fill out this Excel spreadsheet in order to then send it to an admin along with receipts and a bunch of other stuff. Then hopefully we get our money back. No one really likes doing this because it takes a huge amount of time. It's super frustrating. It's kind of very similar if you use other very general purpose apps that happen to be able to do other things that you want. People are really not asking the right question. It's not whether Excel or these systems can do what you want, but whether or not you should. If you think about data analysis, we have a very similar paradigm where for the most part the dominant ways that we interact with the database would be through Excel or some general purpose data exploration application, whereas yes, you can do things that you want to do, but if you have a very specific purpose, then users end up conforming their own use cases and the way they perform analysis to the interfaces rather than the other way around. If you look at the web or mobile, it's kind of the opposite. Instead of Excel spreadsheets, you have to have an expensive one, which does just one thing and one thing very well. It's very easy to use and it's kind of a pleasure. Similarly, instead of opening up Photoshop and using this crazy complicated system, we have very custom filter applications in order to make yourself look more beautiful or the landscape more beautiful. Sometimes if you also have developed specialized applications for interactive visualization, you can also enjoy this kind of simplicity. You want me to use this. Either this or this. One or the other. That is the case. Yes, this is the power of customized interfaces and does it spark joy in people? Yes, Marie Kondo, it does spark joy. Now the question is how do you go about creating these? What are your options? Well, one is of course you can pay people money and hire engineers to figure out what you want and then build an interface and then get something. Of course, money is not raining out of the sky today and so we need some other alternatives. Another thing you might do is to say, well, maybe the end user who knows what they want to do can use a dashboard builder or some other application in order to construct something for themselves. But fundamentally, these interfaces are either very difficult to use or they restrict the kind of analyses you can create in order to keep things simple for end users. Ideally what you would do is you would generate an interface automatically if magic could happen. In fact, in our community, JAG and others have actually done a lot of really interesting work in creating search or crowd type interfaces for databases. It takes a database as input and generates forms such as these for searching or adding new records. Now, they are not analysis specific and so if you have a large database with lots of tables and lots of attributes, it might not be obvious what type of interface to actually create. Now, a very related piece of work is actually visualization recommendation where given data or some previous viz, you want to recommend some other visualization. So the thing that they don't do is actually create the interactive controls which is the focus of our work. Our idea is to why not just go directly with the analysis task in order to generate the interface. That's kind of what we would like to do but it's very hard to know how do you represent and how do you actually get the analysis task. One thing you could do is imagine you could open up someone's brain and try to figure it out from that way but it could be a little bit messy. The approach that we're taking is kind of like a proxy for now is to leverage query logs. These are basically logs of analyses and queries that people have performed in the past already. You can think of them as some representation of at least analyses people have tried to perform. The nice thing is they're pretty much ubiquitous. You could instrument Jupyter notebooks, database recovery logs, provenance is increasingly ubiquitous in everywhere and so on. This already gives us some leverage. The hard work now, I would say the main contribution if you're to skip most of the paper is to define a model to go from a bunch of text that represents the queries on one side into some representation of the set of controls in the interface on the right. I would say that is the primary contribution and then we take a syntactic approach to this. What that means is all we take as input are the queries and the grammar and then we try to generate an interface without looking at the database or having access to database internals. I want to give you an overview of the approach in terms of how we think about this problem. Really our mental model comes from the reverse. If you think about what an interface is, it composes two components. The first is you run a query and then you get the result and you visualise it. Then you have a set of interactive controls that allow you to systematically transform the underlying query in some way. If you imagine transforming it in all possible different ways using these controls, then what that represents is effectively its expressiveness, some set of queries that it can express. What we're doing is the opposite. We're starting with some sample of that and we want to identify essentially the set of widgets that we would need as well as the initial query in order to express these queries. That's what we're going to do. We basically do two steps. One is given these queries, we would like to mine them for interactions and then we would like to map them onto widgets that we can then generate interfaces with. The hard part is going to be generating and identifying these widgets. To start from the left, we model queries as abstract syntax trees. What that means is instead of these string expressions here like average and count, what we're going to do is we're going to parse them into these abstract syntax trees and then look at differences in that space. The nice thing is it then gives us a little bit of semantic information but without having access to the database. This is an abstract syntax tree for a very, very simple SQL query. It can be a little bit complicated. What we would like to do now is mine these interactions. What that means is in this case, there are really two ways that we can go from query one to query two. We can either wholesale swap the entire tree for each other or we can notice that we want to transform average to count and A to B. Both of those are totally valid and both of those are ways that the user could try to express it. What we can think of now is query one and query two constitute two nodes in a graph and then we have different types of colored edges that correspond to the types of transformations in the subtree, in these trees. If we did, so these are correspond to the location in the abstract syntax tree where you're performing the transformation as well as what you're actually transforming. If you do this between different pairs in the query log, then we end up with what we call the interaction graph, where queries correspond to nodes and then these interactions correspond to edges. For example, maybe here we have three queries, Q1, Q2, Q3, where at a particular location we want to swap A to B for the arguments and then maybe later on B to C to go from Q3 to Q2. Now, what we want to do is map this set of edges to a widget that can express these. For example, we might say, well, a dropdown that changes at this particular location and is instantiated with the options A, B, and C might be a useful thing. If the user now clicks on A, then what they're really doing is they're transforming this part of the query to using this particular subtree. Of course, there's many other representations of widgets that you could imagine. For example, you could also use a text box instead. That gives you a little bit more expressive power, but might be a little bit more difficult to use. Clearly, you can see that there's a trade-off that we would like to be able to express as well. What we're going to do as part of this mapping task is we're going to define a library of widgets. The key part is how do you express these widget types? What I mean by that is you can imagine that different widgets have different types of allowable domains. For example, sliders really only make sense if the subtree can be numeric in some nature. Similarly, a dropdown really should be restricted to K subtrees and so on. You can think of these as hard constraints. Whereas, soft constraints now describe these cost functions to say what my preferences might be. It turns out the UI literature and the HCI literature have studied this very, very deeply. We're not going to innovate on this area, except to say that you can encode things like user preference or ease of use and usability and these sorts of metrics into a very simple polynomial cost function. Basically, we do this and we just say we're going to model this as a quadratic equation for each type of widget in order to model how easy it is to use. Given all of these definitions, ultimately what we want to do is given a query log and a library of these widget types, we would like to generate the lowest cost interface as defined by these cost functions that expresses all the queries in the log, meaning that all the queries in the log are reachable in this graph via the widgets that I've selected. It's pretty easy to show that this is a hard problem and we use some heuristics to solve this problem. You can probably come up with pretty good heuristics as well. This hopefully gives you a sense of how we model and think about this problem. What I wanted to do is give you a very short demo to see this in action. This works. This is kind of like Jupyter Notebook except it's this thing called franchise. What we can do in this very, very trivial demo is we can write a query. I can say select these two attributes from an error pollution data set. Here, basically, I can say I want to render as scatter plot. This is what you expect. Now I might say, well, I actually want to see this for particular days. I might want to filter on a day. At this point, if this is the only thing I ever want to do, then there's really only a couple options that I have. Let's say now I say, oh, I want to change the day a little bit. We see that the widgets have changed. You don't have to change widgets like this, but you can. Let's say if I turn it to 30, then we recognize that this is actually the focus point of the analysis, and then now it gives me the slider that I can then now use to vary and stop typing. So this is one use case where as you're performing analysis, you can generate these interfaces for free and then throw them away when you're done with it. But you can imagine other types of use cases as well. All right. Let's see. I want to talk a little bit about experiments. So overall, we ran a bunch of experiments to understand how effective our algorithms are. And to be perfectly frank, it works for arbitrarily complex queries, but the transformations between them need to have some structure that we can exploit. And so that's nice as an alternative to editing and manually typing SQL. So that's nice. So this is kind of an example where this is a very simple query log from SDSS, the Sloan Digital Sky Survey query log, and then this is kind of like the set of widgets that we end up generating. So they seem to be sensible. And of course, it depends. And so if you have ad hoc queries or more complicated transformations, then of course, we're not going to do very well. Now there's kind of like one experiment that I wanted to show you for the sake of time, which is a very short user study that we did. And this kind of is nice because it highlights one kind of like use case where it might be nice to be able to automatically generate interfaces. So here SDSS already has their own web form to perform a bunch of tasks. Essentially of the form, I want to find stars or celestial objects based on a bunch of criteria. And they actually have this document that describes a set of tasks that this web form is well suited for. So what we did was we took a sample of the query log and then we used prison interfaces to generate a set of widgets. Now the layout and everything is manually laid out. And so what you should focus on is just the fact that we've generated a pile of widgets. Now even here, you can notice a couple things. First is there are portions of the original form that we didn't bother to generate. And part of it is because we didn't see evidence of the need to generate those ahead of time. So you could argue it's simplification or that we've kind of lost the ability to do things. Either way, that's kind of what we've done. The other thing is we asked users to perform four of the tasks and to see how well they perform in terms of like time in order to submit the task as well as whether or not it was accurate. For the most part, actually, we don't do better or worse than the original interface except for task one where we did much better. And it turned out the reason was because the original form actually didn't have controls to express this particular task, which is very simple. It's really just looking up objects by their object ID. And turned out that there were other ways, for example, manually typing queries where you can actually do it using the original form, but you end up needing to know the schema and then you know SQL and be able to type things. And so this is kind of like one example where perhaps in many cases you don't actually know upfront when you're actually designing interfaces what the actual analysis you intend to be. And so it'd be nice to dynamically know this after the fact and generate custom interfaces for those. So that's kind of like one possibility how you might use such a system. All right. So I think I have a little bit of time to go into commentary. So basically to come back, the purpose of this work and kind of the sub-area of databases I think about is basically human data interfaces. How do you design interfaces and what is the way that you kind of interact with data? And I would argue that this is actually like fundamentally a very nice data management problem, right? And we have a lot of opportunity to make big contributions in this emerging area called human data interaction. And one way to think about this is in the US we have a couple million software engineers that can do very powerful things and build very powerful tools. But of course that is about 100x smaller than the rest of the population, right? Even if you just look at service workers, there's still 30x people that all want to work with data, right, in banks and insurance companies and governments and so on. And they don't really have this technical expertise. And so the question is what kind of tools can we use to scale out or to kind of like enable existing software engineers or non-technical people to basically scale out and have the disability, right? And so that's something that's worth thinking about. And it is fundamentally a data management problem. It's like how you interact and manage data. And so all I'm saying is outside of efficiency there's many other criteria that are also worthwhile to think about. And so this is a small piece towards what we consider large-scale customized visual analysis interface generation. And our main contribution is this unified model and this way of thinking about it. And of course there's a lot of work to do on our side as well. We don't really think about semantics. We don't think about the database, the data, right? We don't think about generating labels, layout. There's a lot of things we don't do. But this is kind of like the approach that we're going to take and how we're going to move forward. So this is all part of the data visualization management project at Columbia. And so at this point I'm happy to take questions. Thank you. Thanks so much. Goodbye.
perform analysis to the interfaces rather than input and generates forms such as these for query logs. These are basically logs of analyses interface on the right. I would say that is the primary contribution and then we take a syntactic transform the underlying query in some way. If expressiveness, some set of queries that it can between different pairs in the query log, then we cost functions that expresses all the queries in attributes from an error pollution data set. is actually the focus point of the analysis, and can generate these interfaces for free and then experiments to understand how effective our from SDSS, the Sloan Digital Sky Survey query interfaces to generate a set of widgets. Now the users to perform four of the tasks and to see how to express this particular task, which is very simple. It's really just looking up objects by fact and generate custom interfaces for those. So contribution is this unified model and this way like the approach that we're going to take and