400964
Searching the Web
We offer an overview of current Web search engine design. After introducing a generic search engine architecture, we examine each engine component in turn. We cover crawling, local Web page storage, indexing, and the use of link analysis for boosting search performance. The most common design and implementation techniques for each of these components are presented. For this presentation we draw from the literature and from our own experimental search engine testbed. Emphasis is on introducing the fundamental concepts and the results of several performance analyses we conducted to compare different designs.
2425278
Shuffling a Stacked Deck: The Case for Partially Randomized Ranking of Search Engine Results
In-degree, PageRank, number of visits and other measures of Web page popularity significantly influence the ranking of search results by modern search engines. The assumption is that popularity is closely correlated with quality, a more elusive concept that is difficult to measure directly. Unfortunately, the correlation between popularity and quality is very weak for newly-created pages that have yet to receive many visits and/or in-links. Worse, since discovery of new content is largely done by querying search engines, and because users usually focus their attention on the top few results, newly-created but high-quality pages are effectively "shut out," and it can take a very long time before they become popular.We propose a simple and elegant solution to this problem: the introduction of a controlled amount of randomness into search result ranking methods. Doing so offers new pages a chance to prove their worth, although clearly using too much randomness will degrade result quality and annul any benefits achieved. Hence there is a tradeoff between exploration to estimate the quality of new pages and exploitation of pages already known to be of high quality. We study this tradeoff both analytically and via simulation, in the context of an economic objective function based on aggregate result quality amortized over time. We show that a modest amount of randomness leads to improved search results.
171503
Parallel crawlers
In this paper we study how we can design an effective parallel crawler. As the size of the Web grows, it becomes imperative to parallelize a crawling process, in order to finish downloading pages in a reasonable amount of time. We first propose multiple architectures for a parallel crawler and identify fundamental issues related to parallel crawling. Based on this understanding, we then propose metrics to evaluate a parallel crawler, and compare the proposed architectures using 40 million pages collected from the Web. Our results clarify the relative merits of each architecture and provide a good guideline on when to adopt which architecture.
1713822
A fast regular expression indexing engine
In this paper; we describe the design, architecture, and lessons learned from the implementation of a fast regular-expression indexing engine FREE. FREE uses a prebuilt index to identify the text data units which may contain a matching string and only examines these further. In this way, FREE shows orders of magnitude performance improvement in certain cases over standard regular expression matching systems, such as lex, awk and grep.
1739520
Exploiting Geographical Location Information of Web Pages
Many information resources on the web are relevant primarily to limited geographical communities. For instance, web sites containing information on restaurants, theaters, and apartment rentals are relevant primarily to web users in geographical proximity to these locations. In contrast, other information resources are relevant to a broader geographical community. For instance, an on-line newspaper may be relevant to users across the United States. Unfortunately, the geographical scope of web resources is largely ignored by web search engines. We make the case for identifying and exploiting the geographical location information of web sites so that web search engines can rank resources in a geographically sensitive fashion, in addition to using more traditional information-retrieval strategies. In this paper, we first consider how to compute the geographical location of web pages. Subsequently, we consider how to exploit such information in one specific "proof-of-concept" application we implemented in JAVA, and discuss other examples as well.
516682
Efficient Crawling Through URL Ordering
In this paper we study in what order a crawler should visit the URLs it has seen, in order to obtain more "important" pages first. Obtaining important pages rapidly can be very useful when a crawler cannot visit the entire Web in a reasonable amount of time. We define several importance metrics, ordering schemes, and performance evaluation measures for this problem. We also experimentally evaluate the ordering schemes on the Stanford University Web. Our results show that a crawler with a good ordering scheme can obtain important pages significantly faster than one without.
32944410
SPHINX - A Hybrid Knowledge Representation System
29212181
Concurrency and inheritance in actor-based object-oriented languages
Abstract The combination of concurrent and object-oriented programming is a promising paradigm. However, the design of such an integrated language faces a serious problem: concurrent features of the language may interfere with object-oriented features such as inheritance and encapsulation. We present a new concurrency control mechanism for concurrent object-oriented languages based on the actor model, with the aim of avoiding such a problem. The concurrency constraints of an object are expressed with a pair of constructs—behavior abstraction sets and replacement clauses. They are completely separated from the code of methods; moreover, they are the units of inheritance. We introduce a new replacement scheme called lazy replacement for actor-based languages to reduce complexity in expressing the concurrency constraints of objects. We examine the relationship between the proposed concurrency control mechanism and a well-known inheritance mechanism using an experimental language.
361499
Energy Efficient Broadcast in Wireless Ad hoc Networks with Hitch-hiking
In this paper, we propose a novel concept called Hitch-hiking in order to reduce the energy consumption of broadcast application for wireless networks. Hitch-hiking takes advantage of the physical layer design that facilitates the combining of partial signals to obtain the complete information. The concept of combining partial signals using maximal ratio combiner [15] has been used to improve the reliability of the communication link but has never been exploited to reduce energy consumption in broadcasting over wireless ad hoc networks. We study the advantage of Hitch-hiking for the scenario when the transmission power level of nodes is fixed as well as the scenario when the nodes can adjust their power level. For both scenarios, we show that Hitch-hiking is advantageous and have proposed algorithms to construct broadcast tree with Hitch-hiking taken into consideration. For fixed transmission power case, we propose and analyze a centralized heuristic algorithm called SPWMH (Single Power Wireless Multicast with Hitch-hiking) to construct a broadcast tree with minimum forwarding nodes. For the latter case, we propose a centralized heuristic algorithm called Wireless Multicast with Hitch-hiking (WMH) to construct an energy efficient tree using Hitch-hiking and also present a distributed version of the heuristic. We also evaluate the proposed heuristics through simulation. Simulation results show that Hitch-hiking can reduce the transmission cost of broadcast by as much as 50%. Further, we propose and evaluate a protocol called Power Saving with Broadcast Tree (PSBT) that reduces energy consumption of broadcast by eliminating redundancy in receive operation. Finally, we propose an algorithm that takes advantage of both Hitch-hiking and PSBT in conserving energy.
7663373
Performance analysis of multipath multistage interconnection networks with nonuniform output traffic distribution
Develops the analysis of output buffers in multipath multistage interconnection networks. The delay-throughput performance of output buffers in multipath interconnection networks is closely related to the output traffic distribution, which is the distribution of packet arrival rates at each output link connected to a given output module. Many multipath networks produce nonuniform output traffic distributions, even if the input traffic pattern is uniform. We propose in this paper a performance model for output buffer analysis with respect to such a nonuniform output traffic distribution and analyze the output buffer performance of several multipath networks by our proposed analysis model. It is shown that the output traffic distributions are different for the various multipath networks and the output buffer performance (such as packet loss probability and delay) gets better as the nonuniformity of the output traffic distribution becomes higher.
19746677
Performance analysis of an ATM switch with multiple paths
An ATM switch based on multistage interconnection networks with multiple paths can support higher bandwidth than that of buffered networks with single path by passing multiple packets to the same destination simultaneously. These multiple packets are buffered in the output buffer of the destination. The performance of output buffer in the switch with multiple paths is closely dependent on the output traffic distribution, which is the packet arrival rate at each output link destined to a given output port. As the nonuniformity of the output traffic distribution becomes higher, the performance of the output buffer as like delay and packet loss probability gets better. In this paper, we propose a new self-routing switch architecture with multiple paths called Fly network and analyze the performance of the switching network and the output buffer focusing on the output traffic distribution. It is shown that the Fly network supports best throughput and latency of the output buffer by producing a high degree of the nonuniform output traffic distribution.