8707040
Extending SQL for Decision Support Applications
The challenge of extending database systems for decision support applications has been thetopic of much recent research—a very incomplete list of previous work includes [11, 8, 12, 4,10, 5]. Yet, there is no generally accepted solution for the problem, which remains a criticalone, since the inability of current DBMSs to support data mining applications is well-testedand clearly documented [12].Our research approach in addressing this diﬃcult problem is motivated by the observa-tion that aggregate functions provide the linchpin for most decision support computations;moreover inductive discovery from large data sets can be viewed as the process of aggregat-ing low level data into statistical summaries of semantic signiﬁcance. Therefore, the ATLaSsystem designed at UCLA [2] allows end-users to deﬁne new powerful aggregate functionsby writing them in SQL. The same mechanism can be used to deﬁne new table functionsin ATLaS, whose name stands for Aggregate & Table Language and System. ATLaS is thesuccessor of the AXL system described in [15].These SQL-based native extension mechanisms turn ATLaS into a powerful and ﬂexiblesystem for advanced data-intensive applications, including applications from many domainsthat are not supported well by current Object-Relational database systems, which stillsuﬀer from limited extensibility. In fact, the only extensibility mechanism now providedby Object-Relational systems relies on nonnative datablades—i.e., on external functionsdeﬁned in a procedural language and imported into SQL.ATLaS is very eﬀective at expressing decision support tasks: we demonstrate this prop-erty by the eﬃcient implementation of several functions, such as rollups, datacubes, clas-siﬁers, and frequent item sets for association rules [2]. The performance of these functionsexpressed in ATLaS is typically within 30% of the performance of the same algorithmscoded in C/C++. To achieve this level of performance, ATLaS supports various optimiza-tion techniques, and the ability of manipulating in-memory tables in SQL. In fact, attributesof reference type in such tables allow the eﬃcient support of data structures, such as tries,that are instrumental in implementing data mining algorithms, such as Apriori [1, 7].The stream-oriented computation model used by ATLaS contrasts with the computa-tion model based on ‘blocking’ semantics that is normally used for aggregates in currentdatabase systems. Thus, online aggregates [9], time-series queries [13], sliding-window ag-gregates, approximate aggregates, and continuous queries [3] are naturally supported inATLaS. Furthermore, important properties of an ATLaS program, such as blocking behav-ior and monotonicity, can be easily inferred from the syntactic structure of the program [14].
5103963
Adaptive Load Diffusion for Multiway Windowed Stream Joins
In this paper, we present an adaptive load diffusion operator to enable scalable processing of multiway windowed stream joins (MWSJs) using a cluster system. The load diffusion is achieved by a set of novel semantics-pre serving tuple routing algorithms. Different from previous work, the load diffusion operator can (1) preserve the MWSJ semantics while spreading tuples to different hosts for parallel join processing; (2) achieve fine-grained load balancing among distributed hosts; and (3) perform semantics-preserving online adaptations to maintain optimal performance in dynamic stream environments. We have implemented a prototype of the distributed MWSJ framework on top of the System S distributed stream processing system. Our experiment results based on both real data streams and synthetic workloads show that the load diffusion algorithms can efficiently scale-up the performance of MWSJ processing with low overhead.
1498767
Clustering by pattern similarity in large data sets
Clustering is the process of grouping a set of objects into classes of similar objects. Although definitions of similarity vary from one clustering model to another, in most of these models the concept of similarity is based on distances, e.g., Euclidean distance or cosine distance. In other words, similar objects are required to have close values on at least a set of dimensions. In this paper, we explore a more general type of similarity. Under the pCluster model we proposed, two objects are similar if they exhibit a coherent pattern on a subset of dimensions. For instance, in DNA microarray analysis, the expression levels of two genes may rise and fall synchronously in response to a set of environmental stimuli. Although the magnitude of their expression levels may not be close, the patterns they exhibit can be very much alike. Discovery of such clusters of genes is essential in revealing significant connections in gene regulatory networks. E-commerce applications, such as collaborative filtering, can also benefit from the new model, which captures not only the closeness of values of certain leading indicators but also the closeness of (purchasing, browsing, etc.) patterns exhibited by the customers. Our paper introduces an effective algorithm to detect such clusters, and we perform tests on several real and synthetic data sets to show its effectiveness.
14876394
User Defined Aggregates for Logical Data Languages
A new wave of data-intensive and knowledge-based applications|such as data mining and decision support|require the introduction of complex application-speciic aggregate functions. In this paper, we propose extensions for deductive database systems to support these new applications. We develop constructs, formal semantics, and implementation techniques for user-deened aggregates, and describe their realization in an extended LDL++ system recently built at UCLA. With these extensions, the system can support on-line aggregation, roll-ups for data cubing, temporal aggregates for time-series, iceberg queries, and other recently proposed operators used in decision support and data mining procedures. We then discuss the application of this technology to other DBMSs, and in particular to the SQL3 speciications that support the notion of user-deened aggregates. We show that SQL3 suuers from limitations that severely restrict its use in new applications; thus we propose simple extensions similar to those used for LDL++ to overcome such limitations.
5368193
Progressive modeling
Presently, inductive learning is still performed in a frustrating batch process. The user has little interaction with the system and no control over the final accuracy and training time. If the accuracy of the produced model is too low, all the computing resources are misspent. In this paper we propose a progressive modeling framework. In progressive modeling, the learning algorithm estimates online both the accuracy of the final model and remaining training time. If the estimated accuracy is far below expectation, the user can terminate training prior to completion without wasting further resources. If the user chooses to complete the learning process, progressive modeling will compute a model with expected accuracy in expected time. We describe one implementation of progressive modeling using ensemble of classifiers.
2129781
ATLaS: A Native Extension of SQL for Data Mining
A lack of power and extensibility in their query languages has seriously limited the generality of DBMSs and hampered their ability to support new application domains. Considerable efforts by database researchers and commercial DBMS vendors have led to major extensions; yet there remain important applications—particularly data mining—that are not supported well in SQL-3. Thus, there is a pressing need for more general mechanisms for extending SQL and dealing with new application areas, particularly database-centric data mining. To satisfy this need, we allow database users to add new table functions and stream-oriented aggregate functions by defining them in SQL—rather than in external procedural languages as O-R DBMSs currently do. This simple extension turns SQL into a powerful database language, which can express a wide range of applications, including recursive queries, ROLAP aggregates, time-series queries, stream-oriented processing, and data mining functions. In addition to adding great power and flexibility to SQL, these extensions are conducive to performance and data independence. The paper also discusses the system we have developed to support these SQL extensions, and the architecture and techniques used in its realization.
15109806
Design of a next generation sampling service for large scale data analysis applications
Advances in data collection and storage technologies have resulted in large and dynamically growing data sets at many organizations. Database and data mining researchers often use sampling with great effect to scale up performance on these data sets with small cost to accuracy. However, existing techniques often ignore the cost of computing a sample. This cost is often linear in the size of the data set, not the sample, which is expensive. Furthermore, for data mining applications that leverage progressive sampling or bootstrapping-based techniques, this cost can be prohibitive, since they require the generation of multiple samples.To address this problem, we present a solution in the context of a state-of-the-art data analysis center. Specifically, we propose a scalable service that supports sample generation with cost linear in the size of the sample. We then present an efficient parallelization of this service. Our solution leverages high speed interconnects (e.g. Myrinet, Infini-band) for parallel I/O operations with pipelined data transfers. We export an interface that supports both ad-hoc SQL-like querying for database applications, as well as a stand-alone service for data mining applications. We then evaluate our work using queries abstracted from a network monitoring and analysis application, which uses both database and progressive sampling queries. We demonstrate that our implementation achieves good load balance and realizes up to an order of magnitude speedup when compared with extant approaches.
18351320
Spatial data query support in peer-to-peer systems
The distributed hash table (DHT) mechanisms have been proposed to manage data in very large, structured peer-to-peer (P2P) systems. DHT algorithms provide efficient exact-match object search capabilities without requiring global indexing and are hence extremely scalable. However, range queries - which are very common with spatial data - cannot be executed efficiently in these systems because the adoption of uniform hash functions to ensure excellent load balancing unfortunately destroys any existing data locality. In this report we propose a novel technique to preserve spatial locality information while also keeping some of the load balancing properties of DHT based systems. We describe our design as an extension of content-addressable networks (CAN) and illustrate the feasibility of supporting spatial range queries.
9905659
A signaling system using lightweight call sessions
Because of the emergence of heterogeneous access devices and diverse wired and wireless networks, and a substantial lack of support for integrating these networks, we are building a communication network and a service infrastructure that provides integrated telephony and data services across these networks. We generalize the basic call service to support communication between two or more call parties using any number of devices through any media. Call setup has already been addressed by protocols such as the session initiation protocol (SIP), so we focus on a missing component: scalable and fault-tolerant call session maintenance and control after call setup. In this paper, we present a new signaling protocol that offers scalability, high availability, robustness, and flexibility in creating new call processing services. The protocol is compatible with existing call setup protocols, but provides lightweight call session management using a completely decentralized, soft-state group control protocol. We show that this approach simplifies the implementation of the basic call service, including multi-device calling and service handoff between diverse access networks. The design of our protocol follows the principle of separation of control (signaling information) from data. The signaling protocol has been implemented in ICEBERG, an IP-based core network testbed with access to heterogeneous networks.
7829422
Fast Relative Depth Computation for an Active Stereo Vision System
Real-time computation of exact depth is not feasible in an active vision setup. Instead, reliable relative depth information which can be rapidly computed is preferred. In this paper, a stereo cue for computing relative depth obtained from an active stereo vision system is proposed. The proposed stereo cue can be computed purely from the coordinates of points in the stereo pair. The computational cost required is very low. No camera calibration or prior knowledge of the parameters of the stereo vision system is required. We show that the relationship between the relative depth cue and the actual depth in the three-dimensional (3D) space is monotonic. Such a relation is maintained even when the focal length and the vergence angle are changed, so long as the focal lengths of the two cameras are similar. Therefore, real-time implementation in an active vision setup can be realized. Stability analysis shows that the proposed method will be stable in practical situations, unless the stereo camera diverges. Experimental results are presented to highlight the properties and advantages of the proposed method.
611504
Utilizing histogram information
Many commercial database management systems (e.g., DB2, Oracle, etc.) make use of histograms of the value distribution of individual attributes of relations in order to make good selections of query execution plans. These histograms contain partial information about the actual distribution, such as which attribute values occur most frequently, and how often each one occurs, and what value occurs at the kth quantile when the values are sorted.In this paper, we quantitatively assess the information gain (or uncertainty reduction) due to each of these types of histogram information both individually and in combination. Correspondingly, we observe how the accuracy of estimating frequencies of individual values improves with the availability of each of these types of histogram information. We suggest guidelines for constructing histograms tailored to each individual attribute depending on the characteristics of its attribute value distribution.