10966719
The Evolution of the Web and Implications for an Incremental Crawler
In this paper we study how to build an effective incremental crawler. The crawler selectively and incrementally updates its index and/or local collection of web pages, instead of periodically refreshing the collection in batch mode. The incremental crawler can improve the ``freshness'' of the collection significantly and bring in new pages in a more timely manner. We first present results from an experiment conducted on more than half million web pages over 4 months, to estimate how web pages evolve over time. Based on these experimental results, we compare various design choices for an incremental crawler and discuss their trade-offs. We propose an architecture for the incremental crawler, which combines the best design choices.
1739520
Exploiting Geographical Location Information of Web Pages
Many information resources on the web are relevant primarily to limited geographical communities. For instance, web sites containing information on restaurants, theaters, and apartment rentals are relevant primarily to web users in geographical proximity to these locations. In contrast, other information resources are relevant to a broader geographical community. For instance, an on-line newspaper may be relevant to users across the United States. Unfortunately, the geographical scope of web resources is largely ignored by web search engines. We make the case for identifying and exploiting the geographical location information of web sites so that web search engines can rank resources in a geographically sensitive fashion, in addition to using more traditional information-retrieval strategies. In this paper, we first consider how to compute the geographical location of web pages. Subsequently, we consider how to exploit such information in one specific "proof-of-concept" application we implemented in JAVA, and discuss other examples as well.
171503
Parallel crawlers
In this paper we study how we can design an effective parallel crawler. As the size of the Web grows, it becomes imperative to parallelize a crawling process, in order to finish downloading pages in a reasonable amount of time. We first propose multiple architectures for a parallel crawler and identify fundamental issues related to parallel crawling. Based on this understanding, we then propose metrics to evaluate a parallel crawler, and compare the proposed architectures using 40 million pages collected from the Web. Our results clarify the relative merits of each architecture and provide a good guideline on when to adopt which architecture.
1713822
A fast regular expression indexing engine
In this paper; we describe the design, architecture, and lessons learned from the implementation of a fast regular-expression indexing engine FREE. FREE uses a prebuilt index to identify the text data units which may contain a matching string and only examines these further. In this way, FREE shows orders of magnitude performance improvement in certain cases over standard regular expression matching systems, such as lex, awk and grep.
516682
Efficient Crawling Through URL Ordering
In this paper we study in what order a crawler should visit the URLs it has seen, in order to obtain more "important" pages first. Obtaining important pages rapidly can be very useful when a crawler cannot visit the entire Web in a reasonable amount of time. We define several importance metrics, ordering schemes, and performance evaluation measures for this problem. We also experimentally evaluate the ordering schemes on the Stanford University Web. Our results show that a crawler with a good ordering scheme can obtain important pages significantly faster than one without.
1720833
Automatic identification of user interest for personalized search
One hundred users, one hundred needs. As more and more topics are being discussed on the web and our vocabulary remains relatively stable, it is increasingly difficult to let the search engine know what we want. Coping with ambiguous queries has long been an important part of the research on Information Retrieval, but still remains a challenging task. Personalized search has recently got significant attention in addressing this challenge in the web search community, based on the premise that a user's general preference may help the search engine disambiguate the true intention of a query. However, studies have shown that users are reluctant to provide any explicit input on their personal preference. In this paper, we study how a search engine can learn a user's preference automatically based on her past click history and how it can use the user preference to personalize search results. Our experiments show that users' preferences can be learned accurately even from little click-history data and personalized search based on user preference yields significant improvements over the best existing ranking mechanism in the literature.
19746677
Performance analysis of an ATM switch with multiple paths
An ATM switch based on multistage interconnection networks with multiple paths can support higher bandwidth than that of buffered networks with single path by passing multiple packets to the same destination simultaneously. These multiple packets are buffered in the output buffer of the destination. The performance of output buffer in the switch with multiple paths is closely dependent on the output traffic distribution, which is the packet arrival rate at each output link destined to a given output port. As the nonuniformity of the output traffic distribution becomes higher, the performance of the output buffer as like delay and packet loss probability gets better. In this paper, we propose a new self-routing switch architecture with multiple paths called Fly network and analyze the performance of the switching network and the output buffer focusing on the output traffic distribution. It is shown that the Fly network supports best throughput and latency of the output buffer by producing a high degree of the nonuniform output traffic distribution.
39501622
A logic programming approach to hybrid knowledge representation
A new hybrid system called Sphinx is presented which combines classification-based reasoning with logic programming technique. It consists of two major components: a terminological component and an assertional one. The former uses a frame-based knowledge representation scheme based on classification and the latter is a theorem prover based on Horn logic. The assertional component can also handle a first-order query language and wh-questions. A new knowledge base maintenance mechanism based on the negation as failure inference rule enables Sphinx to support incremental assertions and retractions. In addition, Sphinx provides explanation capability to help the user in developing and debugging knowledge bases.
29212181
Concurrency and inheritance in actor-based object-oriented languages
Abstract The combination of concurrent and object-oriented programming is a promising paradigm. However, the design of such an integrated language faces a serious problem: concurrent features of the language may interfere with object-oriented features such as inheritance and encapsulation. We present a new concurrency control mechanism for concurrent object-oriented languages based on the actor model, with the aim of avoiding such a problem. The concurrency constraints of an object are expressed with a pair of constructs—behavior abstraction sets and replacement clauses. They are completely separated from the code of methods; moreover, they are the units of inheritance. We introduce a new replacement scheme called lazy replacement for actor-based languages to reduce complexity in expressing the concurrency constraints of objects. We examine the relationship between the proposed concurrency control mechanism and a well-known inheritance mechanism using an experimental language.
9811102
Four-bend top-down global routing
We propose a new global net distribution approach for high-performance m/spl times/m two-dimensional arrays of very large scale integration and multichip-modules. The objective is to route n nets with minimum density of global cells, using a "small" number of bends. There are a number of applications where it is necessary to limit the number of bends on each wire. For example, it is desirable to limit the number of bends on each microstrip (transmission) line, for mismatches of line impedance can cause reflections from the junction points such as bends and vias. Furthermore, for high-performance routing, intersections of wires cause the use of more vias, which in turn require the use of more routing resources (because of the larger via pitch). This is the first paper that addresses a graph-theoretic framework to solve the bend-constrained global routing problem in two-dimensional arrays. In this paper, at each level of an underlying quad-tree, we present a novel four-bend routing algorithm by decomposing the original problem at level i into two subproblems that can be solved exactly based on a two-stage approach of smaller-sized linear program followed by min-cost flow networks. The overall (i.e., entire level of the four-way partition hierarchy) constraint and variable size for the first stage is O(md/sub o/), while the overall run time for the second stage is O(n/sup 3/ log n/sup 2/). The time complexity of such a hierarchical approach is one order of magnitude less than one of constructing a global routing using the min-cost-flow-based flat design approach. Last, we present an extension that permits a limited degree of control over the number of bends. The proposed algorithm can also be used for estimating the wireability in the early design planning stage for high-level synthesis. Experimental results showed the effectiveness of the proposed algorithm.
30800683
An optimal signal design for band-limited asynchronous DS-CDMA Communications
An optimal signal design for band-limited, asynchronous, direct-sequence code-division multiple-access (DS-CDMA) communications with aperiodic random spreading sequences and a conventional matched filter receiver is considered in an additive white Gaussian noise (AWGN) channel. With bandwidth defined in the strict sense, two optimization problems are solved under finite bandwidth and zero interchip interference constraints. First, a chip waveform optimization is performed given the system bandwidth, the data symbol transmission rate, and the processing gain. A technique to characterize a band-limited chip waveform with a finite number of parameters is developed, and it is used to derive optimum chip waveforms which minimize the effect of multiple-access interference (MAI) for any energy and delay profile of users. Next, a joint optimization of the processing gain and the chip waveform is performed, given the system bandwidth and the data symbol transmission rate. A sufficient condition for a system to have lower average probability of bit error for any energy profile is found, and it is used to derive some design strategies. It is shown that the flat spectrum pulse with the processing gain leading to zero excess bandwidth results in the minimum average probability of bit error. Design examples and numerical results are also provided.