9362566
Estimating frequency of change
Many online data sources are updated autonomously and independently. In this article, we make the case for estimating the change frequency of data to improve Web crawlers, Web caches and to help data mining. We first identify various scenarios, where different applications have different requirements on the accuracy of the estimated frequency. Then we develop several "frequency estimators" for the identified scenarios, showing analytically and experimentally how precise they are. In many cases, our proposed estimators predict change frequencies much more accurately and improve the effectiveness of applications. For example, a Web crawler could achieve 35% improvement in "freshness" simply by adopting our proposed estimator.
2425278
Shuffling a Stacked Deck: The Case for Partially Randomized Ranking of Search Engine Results
In-degree, PageRank, number of visits and other measures of Web page popularity significantly influence the ranking of search results by modern search engines. The assumption is that popularity is closely correlated with quality, a more elusive concept that is difficult to measure directly. Unfortunately, the correlation between popularity and quality is very weak for newly-created pages that have yet to receive many visits and/or in-links. Worse, since discovery of new content is largely done by querying search engines, and because users usually focus their attention on the top few results, newly-created but high-quality pages are effectively "shut out," and it can take a very long time before they become popular.We propose a simple and elegant solution to this problem: the introduction of a controlled amount of randomness into search result ranking methods. Doing so offers new pages a chance to prove their worth, although clearly using too much randomness will degrade result quality and annul any benefits achieved. Hence there is a tradeoff between exploration to estimate the quality of new pages and exploitation of pages already known to be of high quality. We study this tradeoff both analytically and via simulation, in the context of an economic objective function based on aggregate result quality amortized over time. We show that a modest amount of randomness leads to improved search results.
516682
Efficient Crawling Through URL Ordering
In this paper we study in what order a crawler should visit the URLs it has seen, in order to obtain more "important" pages first. Obtaining important pages rapidly can be very useful when a crawler cannot visit the entire Web in a reasonable amount of time. We define several importance metrics, ordering schemes, and performance evaluation measures for this problem. We also experimentally evaluate the ordering schemes on the Stanford University Web. Our results show that a crawler with a good ordering scheme can obtain important pages significantly faster than one without.
1739520
Exploiting Geographical Location Information of Web Pages
Many information resources on the web are relevant primarily to limited geographical communities. For instance, web sites containing information on restaurants, theaters, and apartment rentals are relevant primarily to web users in geographical proximity to these locations. In contrast, other information resources are relevant to a broader geographical community. For instance, an on-line newspaper may be relevant to users across the United States. Unfortunately, the geographical scope of web resources is largely ignored by web search engines. We make the case for identifying and exploiting the geographical location information of web sites so that web search engines can rank resources in a geographically sensitive fashion, in addition to using more traditional information-retrieval strategies. In this paper, we first consider how to compute the geographical location of web pages. Subsequently, we consider how to exploit such information in one specific "proof-of-concept" application we implemented in JAVA, and discuss other examples as well.
171503
Parallel crawlers
In this paper we study how we can design an effective parallel crawler. As the size of the Web grows, it becomes imperative to parallelize a crawling process, in order to finish downloading pages in a reasonable amount of time. We first propose multiple architectures for a parallel crawler and identify fundamental issues related to parallel crawling. Based on this understanding, we then propose metrics to evaluate a parallel crawler, and compare the proposed architectures using 40 million pages collected from the Web. Our results clarify the relative merits of each architecture and provide a good guideline on when to adopt which architecture.
1713822
A fast regular expression indexing engine
In this paper; we describe the design, architecture, and lessons learned from the implementation of a fast regular-expression indexing engine FREE. FREE uses a prebuilt index to identify the text data units which may contain a matching string and only examines these further. In this way, FREE shows orders of magnitude performance improvement in certain cases over standard regular expression matching systems, such as lex, awk and grep.
36016648
Echo Watermarking in Sub-band Domain
Echo hiding is a method for embedding information, called watermark, into an audio signal. In general, echo hiding is processed in the time domain by convolving audio signal with echo kernel, without any consideration of the frequency characteristics. In this paper, we propose an echo hiding technique, which inserts echo into sub-band signal so as to take account into the frequency characteristics. The proposed echo hiding enables to embed high-energy echo, while minimizing the host audio quality distortion. In addition, the proposed allows increasing watermark capacity, since it is possible to embed simultaneously some watermark bits into different sub-band signals. The simulation results show that the proposed is more effective than the conventional echo hiding, in terms of watermark capacity and robustness against various attacks.
27932771
Novel algorithms for multicast communication in self-routing MIN-based ATM switches
We discuss the multicast communication in the self-routing multistage interconnection network (MIN) for constructing the internal architecture of asynchronous transfer mode (ATM) switches. Many applications of ATM switches require multicast communications in addition to conventional point-to-point communications. This paper presents a novel approach to supporting multicast communication, on the basis of the recursive scheme that recycles a multicast packet one or more times through the network to reach at desired destinations. We also propose cost-effective multicast algorithms providing deadlock-freedom in MIN-based ATM switches. The proposed algorithms require a small and fixed number of recycling passes and a reasonable number of links used. The proposed algorithms can be easily applicable to buffered MIN-based ATM switches.
45001657
A New Subsumption Method in the Connection Graph Proof Procedure
Compared to most of the other resolution-based proof procedures, the connection graph proof procedure (CGPP) is superior due to its effective search space reduction rules. A test for subsumption in CGPP is sufficiently efficient in comparison with the standard test. But the subsumption with no restriction may cause logical inconsistency. The restriction on subsumption to maintain the consistency has been an open problem since it was first noticed in 1981. We define a newθ-subsumption which maintains the logical consistency and provides the graph transformation method which makes C subsume D with the new definition if C subsumes D with the original definition ofθ-sumsumption. 
 
In the subsumption test in CGPP, it is essential to find a set of pairwise strongly compatible matching substitutions between literals in two clauses. This paper presents an algorithm which finds a set of pairwise strongly compatible matching substitutions efficiently, based on both the reduction of possible matching substitutions and bit operations. The worst-case time complexity of this algorithm is analyzed in terms of the estimated maximal number of string comparisons and it is shown that our complexity is lower than the other algorithms based on the s-link test. 
 
Based on the new definition of subsumption and the improved algorithm for the pairwise strongly compatible test, an unrestricted subsumption algorithm and a restricted subsumption algorithm are suggested.
7663373
Performance analysis of multipath multistage interconnection networks with nonuniform output traffic distribution
Develops the analysis of output buffers in multipath multistage interconnection networks. The delay-throughput performance of output buffers in multipath interconnection networks is closely related to the output traffic distribution, which is the distribution of packet arrival rates at each output link connected to a given output module. Many multipath networks produce nonuniform output traffic distributions, even if the input traffic pattern is uniform. We propose in this paper a performance model for output buffer analysis with respect to such a nonuniform output traffic distribution and analyze the output buffer performance of several multipath networks by our proposed analysis model. It is shown that the output traffic distributions are different for the various multipath networks and the output buffer performance (such as packet loss probability and delay) gets better as the nonuniformity of the output traffic distribution becomes higher.
46360239
A New Algorithm for Subsumption Test