2564021
Database research at Columbia University
Columbia University has a number of projects that touch on database systems issues. In this report, we describe the Columbia Fast Query Project (Section 2), the JAM project (Section 3), the CARDGIS project (Section 4), the Columbia Internet Information Searching Project (Section 5), the Columbia Content-Based Visual Query project (Section 6), and projects associated with Columbia’s Programming Systems Laboratory (Section 7).
16656102
Groupwise Processing of Relational Queries
In this paper, we define and examine a particular class of queries called group queries. Group queries are natural queries in many decisionsupport applications. The main characteristic of a group query is that it can be executed in a groupby-group fashion. In other words, the underlying relation(s) can be partitioned (based on some set of attributes) into disjoint groups, and each group can be processed separately. We give a syntactic criterion to identify these queries and prove its sufficiency. We also prove the strong result that every group query has an equivalent formulation that satisfies our syntactic criterion. We describe a general evaluation technique for group queries, and demonstrate how an optimizer can determine this plan. We then consider more complex queries whose components are group queries with potentially different partitioning attributes. We give two methods to identify group query components within such a query. We also give some performance results for group queries expressed in standard SQL, comparing a commercial database system with our optimized plan on top of the same commercial system. These results indicate that there are significant potential performance improvements.
7567374
Foundations of Aggregation Constraints
We introduce a new constraint domain, aggregation constraints, which is useful in database query languages, and in constraint logic programming languages that incorporate aggregate functions. We study the fundamental problem of checking if a conjunction of aggregation constraints is solvable, and present undecidability results for many different classes of aggregation constraints. We describe a complete and minimal axiomatization of the class of aggregation constraints over finite multisets of reals, which permits a natural reduction from the class of aggregation constraints to the class of mixed integer/real, non-linear arithmetic constraints. We then present a polynomial-time algorithm that directly checks for solvability of a useful class of aggregation constraints, where the reduction-based approach does not lead to efficient checks for solvability.
38852147
On Negation in HiLog
Abstract The logic HiLog of Chen, Kifer, and Warren has a second order syntax, while its semantics is first order. HiLog programs with negative literals in the body are considered. A stable-model semantics and a well-founded semantics for this class of program are defined, and it is shown that these semantics generalize the stable-model semantics and the well-founded semantics, respectively, for range-restricted normal programs. A second order property called preservation under extensions is proposed and investigated. Preservation under extensions ensures that the semantics of a program is not changed when rules having no symbols in common with the program are appended to the program. It is shown that for normal programs, domain independence and preservation under extensions are equivalent, while for HiLog programs, preservation under extensions is strictly stronger. Range restrictedness is generalized to HiLog programs in two ways, and it is shown that range restricted HiLog programs are preserved under extensions with respect to the well-founded semantics. Conditions under which the well-founded semantics is two-valued are investigated, and the class of modularly stratified programs is generalized to HiLog. An extension of magic-set techniques to modularly stratified HiLog programs is described.
16955834
Querying Multiple Features of Groups in Relational Databases
Some aggregate and grouping queries are conceptually simple, but difficult to express in SQL. This difficulty causes both conceptual and implementation problems for the SQLbased database system. Complicated queries and views are hard to understand and maintain. Further, the code produced is sometimes unnecessarily inefficient, as we demonstrate experimentally using a commercial database system. In this paper, we examine a class of queries involving (potentially repeated) selection, grouping and aggregation over the same groups, and propose an extension of SQL syntax that allows the succinct representation of these queries. We propose a new relational algebra operation that represents several levels of aggregation over the same groups in an operand relation. We demonstrate that the extended relational operator can be evaluated using efficient algorithms. We describe a translation from the extended SQL language into our algebraic language. We have implemented a preprocessor that evaluates our extended language on top of a commercial l This research was supported by a grant from the AT&T Foundation, by a David and Lucile Packard Foundation Fellowship in Science and Engineering, by a Sloan Foundation Fellowship, by NSF grants IRI-9209029, CDA-90-24735, and by an NSF Young Investigator award. Permission to copy without fee all or part of this material is gmnted provided that the wpier are not made or distributed for direct commercial advantage, the VLDB wpyrinht notice and the title of the publication and its date appear, and notice is given that copying is by permission of the Very Large Data Base Endowment. To copy otherwise, or to republish, requires a fee and/or special permission from the Endowment. Proceedings of the 22nd VLDB Conference Mumbai(Bombay), India, 1996 database system. We demonstrate that on a variety of examples, our implementation improves performance over standard SQL representations of the same examples by orders of magnitude.
31185043
Inferring negative information from disjunctive databases
We propose criteria that any rule for inferring negative information from disjunctive databases should satisfy, and examine existing rules from this viewpoint. We then present a new inference rule, the ‘disjunctive database rule’ (DDR), and compare it to the existing rules with respect to the criteria. In particular, the DDR is equivalent to the CWA for definite databases, it infers no more negative information than the GCWA, and it interprets disjunction inclusively rather than exclusively. We generalize the DDR to a class of layered databases, describe an implementation of the DDR, ‘negation as positive failure’, and study its soundness and completeness properties.
6259994
Hierarchical Peer-To-Peer Systems
Structured peer-to-peer (P2P) lookup services organize peers into a flat overlay network and offer distributed hash table (DHT) functionality. Data is associated with keys and each peer is responsible for a subset of the keys. In hierarchical DHTs, peers are organized into groups, and each group has its autonomous intra-group overlay network and lookup service. Groups are organized in a top-level overlay network. To find a peer that is responsible for a key, the top-level overlay first determines the group responsible for the key; the responsible group then uses its intra-group overlay to determine the specific peer that is responsible for the key. We provide a general framework for hierarchical DHTs with scalable overlay management. We specifically study a two-tier hierarchy that uses Chord for the top level. Our analysis shows that by using the most reliable peers in the top level, the hierarchical design significantly reduces the expected number of hops. We also present a method to construct hierarchical DHTs that map well to the Internet topology and achieve short intra-group communication delay. The results demonstrate the feasibility of locality-based peer groups, which allow P2P systems to take full advantage of the hierarchical design.
45021908
Video-on-Demand Over ATM: Constant-Rate Transmission and Transport
We introduce a specific transport and transmission scheme for video-on-demand (VoD) called constant-rate transmission and transport (CRTT). CRTT establishes a constant bit-rate (CBR) virtual channel between the video provider and the viewer's set-top box (STB) and then transmits cells from the provider into this channel at a constant rate. Since we assume that the number of cells in a frame is variable, CRTT requires that some number of cells be built up in an STB buffer before the commencement of playback. The build up, cell transmission rate, and the set-top memory size must be chosen so that there is no starvation or overflow at the STB. We develop fundamental relationships between these parameters for viable CRTT. We then apply the theory to an MPEG encoding of Star Wars and find that the minimal STB memory far CRTT is 23 Mbytes. We also consider varying the constant rate over a small number of intervals. We find, for example, that for Star Wars approximately 2 Mbytes of set-top memory suffices with 32 constant-rate intervals.
986287
A probability collectives approach to weighted clustering algorithms for ad hoc networks
The Weighted Clustering Algorithm has been proposed for mobile ad hoc wireless networks to select a subset of the nodes to be local purveyors of media access control services to the rest of the nodes. The nodes selected are called clusterheads, and each clusterhead services the other nodes within its radio range. The underlying optimal assignment problem is NP-hard. We introduce a new metaheuristic known as probability collectives to the problem of assigning clusterheads. This is an agent-based approach, with each node in the network acting in a quasi-autonomous way. This new approach shows improved clustering results, particularly with regard to the distribution and balance of cluster sizes.
10446891
Periodic broadcasting with VBR-encoded video
Abstract.We consider designing near video on demand (NVOD) systems that minimize startup latency while maintaining high image quality. Recently nonuniform segmentation has been used to develop periodic broadcasting techniques for NVOD. These techniques give significant reductions in startup latency as compared with more conventional uniform segmentation. Essentially all of these schemes assume, however, that the videos are CBR-encoded. Since a CBR-encoded video has a larger average rate than an open-loop VBR encoding with the same image quality, there is potential to obtain further performance improvements by using VBR video. In this paper we develop a series of multiplexing schemes for the periodic broadcasting of VBR-encoded video that are based on smoothing, server buffering, and client prefetching. Two key but conflicting performance measures exist when using VBR video: latency and packet loss. By introducing small additional delays in our multiplexing schemes, our trace-based numerical work shows that the schemes can achieve nearly 100% link utilization with negligible packet loss. When the ratio of the CBR rate to the VBR average rate is a modest 1.8, startup latency can be reduced by a factor of four or more for common scenarios.
10455339
Pollution in P2P file sharing systems
One way to combat P2P file sharing of copyrighted content is to deposit into the file sharing systems large volumes of polluted files. Without taking sides in the file sharing debate, in this paper we undertake a measurement study of the nature and magnitude of pollution in the FastTrack P2P network, currently the most popular P2P file sharing system. We develop a crawling platform which crawls the majority of the FastTrack Network's 20,000+ supernodes in less than 60 minutes. From the raw data gathered by the crawler for popular audio content, we obtain statistics on the number of unique versions and copies available in a 24-hour period. We develop an automated procedure to detect whether a given version is polluted or not, and we show that the probabilities of false positives and negatives of the detection procedure are very small. We use the data from the crawler and our pollution detection algorithm to determine the fraction of versions and fraction of copies that are polluted for several recent and old songs. We observe that pollution is pervasive for recent popular songs. We also identify and describe a number of anti-pollution mechanisms.