2425278
Shuffling a Stacked Deck: The Case for Partially Randomized Ranking of Search Engine Results
In-degree, PageRank, number of visits and other measures of Web page popularity significantly influence the ranking of search results by modern search engines. The assumption is that popularity is closely correlated with quality, a more elusive concept that is difficult to measure directly. Unfortunately, the correlation between popularity and quality is very weak for newly-created pages that have yet to receive many visits and/or in-links. Worse, since discovery of new content is largely done by querying search engines, and because users usually focus their attention on the top few results, newly-created but high-quality pages are effectively "shut out," and it can take a very long time before they become popular.We propose a simple and elegant solution to this problem: the introduction of a controlled amount of randomness into search result ranking methods. Doing so offers new pages a chance to prove their worth, although clearly using too much randomness will degrade result quality and annul any benefits achieved. Hence there is a tradeoff between exploration to estimate the quality of new pages and exploitation of pages already known to be of high quality. We study this tradeoff both analytically and via simulation, in the context of an economic objective function based on aggregate result quality amortized over time. We show that a modest amount of randomness leads to improved search results.
1089297
Automatic identification of user goals in Web search
There has been recent interests in studying the "goal" behind a user's Web query, so that this goal can be used to improve the quality of a search engine's results. Previous studies have mainly focused on using manual query-log investigation to identify Web query goals. In this paper we study whether and how we can automate this goal-identification process. We first present our results from a human subject study that strongly indicate the feasibility of automatic query-goal identification. We then propose two types of features for the goal-identification task: user-click behavior and anchor-link distribution. Our experimental evaluation shows that by combining these features we can correctly identify the goals for 90% of the queries studied.
10966719
The Evolution of the Web and Implications for an Incremental Crawler
In this paper we study how to build an effective incremental crawler. The crawler selectively and incrementally updates its index and/or local collection of web pages, instead of periodically refreshing the collection in batch mode. The incremental crawler can improve the ``freshness'' of the collection significantly and bring in new pages in a more timely manner. We first present results from an experiment conducted on more than half million web pages over 4 months, to estimate how web pages evolve over time. Based on these experimental results, we compare various design choices for an incremental crawler and discuss their trade-offs. We propose an architecture for the incremental crawler, which combines the best design choices.
5658625
Finding replicated Web collections
Many web documents (such as JAVA FAQs) are being replicated on the Internet. Often entire document collections (such as hyperlinked Linux manuals) are being replicated many times. In this paper, we make the case for identifying replicated documents and collections to improve web crawlers, archivers, and ranking functions used in search engines. The paper describes how to efficiently identify replicated documents and hyperlinked document collections. The challenge is to identify these replicas from an input data set of several tens of millions of web pages and several hundreds of gigabytes of textual data. We also present two real-life case studies where we used replication information to improve a crawler and a search engine. We report these results for a data set of 25 million web pages (about 150 gigabytes of HTML data) crawled from the web.
1713822
A fast regular expression indexing engine
In this paper; we describe the design, architecture, and lessons learned from the implementation of a fast regular-expression indexing engine FREE. FREE uses a prebuilt index to identify the text data units which may contain a matching string and only examines these further. In this way, FREE shows orders of magnitude performance improvement in certain cases over standard regular expression matching systems, such as lex, awk and grep.
398600
What's new on the web?: the evolution of the web from a search engine perspective
We seek to gain improved insight into how Web search engines shouldcope with the evolving Web, in an attempt to provide users with themost up-to-date results possible. For this purpose we collectedweekly snapshots of some 150 Web sites over the course of one year,and measured the evolution of content and link structure. Our measurements focus on aspects of potential interest to search engine designers: the evolution of link structure over time, the rate ofcreation of new pages and new distinct content on the Web, and the rate of change of the content of existing pages under search-centric measures of degree of change.Our findings indicate a rapid turnover rate of Web pages, i.e.,high rates of birth and death, coupled with an even higher rate ofturnover in the hyperlinks that connect them. For pages that persistover time we found that, perhaps surprisingly, the degree of contentshift as measured using TF.IDF cosine distance does not appear to beconsistently correlated with the frequency of contentupdating. Despite this apparent non-correlation, the rate of content shift of a given page is likely to remain consistent over time. That is, pages that change a great deal in one week will likely change by a similarly large degree in the following week. Conversely, pages that experience little change will continue to experience little change. We conclude the paper with a discussion of the potential implications ofour results for the design of effective Web search engines.
30800683
An optimal signal design for band-limited asynchronous DS-CDMA Communications
An optimal signal design for band-limited, asynchronous, direct-sequence code-division multiple-access (DS-CDMA) communications with aperiodic random spreading sequences and a conventional matched filter receiver is considered in an additive white Gaussian noise (AWGN) channel. With bandwidth defined in the strict sense, two optimization problems are solved under finite bandwidth and zero interchip interference constraints. First, a chip waveform optimization is performed given the system bandwidth, the data symbol transmission rate, and the processing gain. A technique to characterize a band-limited chip waveform with a finite number of parameters is developed, and it is used to derive optimum chip waveforms which minimize the effect of multiple-access interference (MAI) for any energy and delay profile of users. Next, a joint optimization of the processing gain and the chip waveform is performed, given the system bandwidth and the data symbol transmission rate. A sufficient condition for a system to have lower average probability of bit error for any energy profile is found, and it is used to derive some design strategies. It is shown that the flat spectrum pulse with the processing gain leading to zero excess bandwidth results in the minimum average probability of bit error. Design examples and numerical results are also provided.
3145493
KPSP: A Knowledge Programming System based on Prolog
45001657
A New Subsumption Method in the Connection Graph Proof Procedure
Compared to most of the other resolution-based proof procedures, the connection graph proof procedure (CGPP) is superior due to its effective search space reduction rules. A test for subsumption in CGPP is sufficiently efficient in comparison with the standard test. But the subsumption with no restriction may cause logical inconsistency. The restriction on subsumption to maintain the consistency has been an open problem since it was first noticed in 1981. We define a newθ-subsumption which maintains the logical consistency and provides the graph transformation method which makes C subsume D with the new definition if C subsumes D with the original definition ofθ-sumsumption. 
 
In the subsumption test in CGPP, it is essential to find a set of pairwise strongly compatible matching substitutions between literals in two clauses. This paper presents an algorithm which finds a set of pairwise strongly compatible matching substitutions efficiently, based on both the reduction of possible matching substitutions and bit operations. The worst-case time complexity of this algorithm is analyzed in terms of the estimated maximal number of string comparisons and it is shown that our complexity is lower than the other algorithms based on the s-link test. 
 
Based on the new definition of subsumption and the improved algorithm for the pairwise strongly compatible test, an unrestricted subsumption algorithm and a restricted subsumption algorithm are suggested.
36016648
Echo Watermarking in Sub-band Domain
Echo hiding is a method for embedding information, called watermark, into an audio signal. In general, echo hiding is processed in the time domain by convolving audio signal with echo kernel, without any consideration of the frequency characteristics. In this paper, we propose an echo hiding technique, which inserts echo into sub-band signal so as to take account into the frequency characteristics. The proposed echo hiding enables to embed high-energy echo, while minimizing the host audio quality distortion. In addition, the proposed allows increasing watermark capacity, since it is possible to embed simultaneously some watermark bits into different sub-band signals. The simulation results show that the proposed is more effective than the conventional echo hiding, in terms of watermark capacity and robustness against various attacks.
29923826
A Relational Dataflow Database Machine Based on Hierarchical Ring Network