1987744
Clustering High Dimensional Massive Scientific Datasets
Many scientific applications can benefit from an efficient clustering algorithm of massively large high dimensional datasets. However most of the developed algorithms are impractical to use when the amount of data is very large. Given N objects each defined by an M-dimensional feature vector, any clustering technique for handling very large datasets in high dimensional space should run in time O(MN) at best, and O(MN log N) in the worst case, using no more than O(MN) storage, for it to be practical. We introduce a hybrid algorithm, called HyCeltyc, for clustering massively large high dimensional datasets in O(MN) time which is linear in the size of the data. HyCeltyc, which stands for HybridCell Density Clustering method, combines a cell-density based algorithm with a hierarchical agglomerative method to identify clusters in linear time. The main steps of the algorithm involve sampling, dimensionality reduction, selection of significant features on which to cluster the data and a grid-based clustering algorihm that is linear in the data size.
14754194
Enabling soft queries for data retrieval
Data retrieval finding relevant data from large databases - has become a serious problem as myriad databases have been brought online in the Web. For instance, querying the for-sale houses in Chicago from realtor.com returns thousands of matching houses. Similarly, querying ''digital camera'' in froogle.com returns hundreds of thousand of results. This data retrieval is essentially an online ranking problem, i.e., ranking data results according to the user's preference effectively and efficiently. This paper proposes a new rank query framework, for effectively incorporating ''user-friendly'' rank-query formulation into ''data base (DB)-friendly'' rank-query processing, in order to enable ''soft'' queries on databases. Our framework assumes, as the ''back-end,'' the score-based ranking model for expressive and efficient query processing. On top of the score-based model, as the ''front-end,'' we adopt an SVM-ranking mechanism for providing intuitive and exploratory query formulation. In essence, our framework enables users to formulate queries simply by ordering some sample objects, while learning the ''DB-friendly'' ranking function F from the partial orders. Such learned functions can then be processed and optimized by existing database systems. We demonstrate the efficiency and effectiveness of our framework using real-life user queries and datasets: our results show that the system effectively learns quantitative ranking functions from qualitative feedback from users with efficient online processing.
28585519
Personal Workflow Management in Support of Pervasive Computing
With the advent of new wireless technologies, mobile personal computers (MPCs) are quickly gaining their popularity. Concurrently an increasingly number of services for MPCs have been developed. However, current services view PMCs as a vehicle to perform individual activities. No attempt has been made to provide process services that reach mobile users via their PMCs. In the real world, many personal applications are process-oriented, e.g., planning a trip, holding a party, applying a new job, etc. We expect PMCs to help manage and execute these processes in the future, serving the role of planning, coordinating, and reminding task executions so as to accomplish predefined personal goals. On the other hand, traditional workflow management systems are mainly used to coordinate business processes in enterprises. These processes must be repetitive and have well-formed structures. Personal processes differ from their business counterparts in that they are subject to change and rigid regulations on control flow are seldom imposed. As a result, personal activities are related mainly due to their data dependencies and their coordination must be flexible. These differences demand a new personal workflow model as well as the innovative design of a personal workflow system (PWFS).
13958377
Myriad: Design and Implementation of a Federated Database Prototype
SUMMARY A key problem in providing ‘enterprise-wide’ information is the integration of databases that have been independently developed. An important requirement is to accommodate heterogeneity and maintain the autonomy of component databases. Myriad is a federated database prototype developed at the University of Minnesota, to provide a testbed for investigating alternatives in architecture and algorithms for database integration, query processing and optimization, and concurrency control and recovery. The system incorporates our group’s research results in these areas. This paper describes our experiences in the design and implementation of Myriad, and in the project management. Special emphasis is given to discussing design alternatives and their impact on Myriad. This paper also presents the software engineering principles and the project management techniques we used in developing Myriad and the lessons we learned. We believe these lessons would be useful for practitioners who wish to develop a similar system. Handling heterogeneity and autonomy were prime objectives throughout the prototyping effort. We are convinced that a prototype federated database is an important infrastructural requirement for the overall goal of ‘enterprise-integration’, and believe Myriad to be a significant contribution towards this.