171503
Parallel crawlers
In this paper we study how we can design an effective parallel crawler. As the size of the Web grows, it becomes imperative to parallelize a crawling process, in order to finish downloading pages in a reasonable amount of time. We first propose multiple architectures for a parallel crawler and identify fundamental issues related to parallel crawling. Based on this understanding, we then propose metrics to evaluate a parallel crawler, and compare the proposed architectures using 40 million pages collected from the Web. Our results clarify the relative merits of each architecture and provide a good guideline on when to adopt which architecture.
1713822
A fast regular expression indexing engine
In this paper; we describe the design, architecture, and lessons learned from the implementation of a fast regular-expression indexing engine FREE. FREE uses a prebuilt index to identify the text data units which may contain a matching string and only examines these further. In this way, FREE shows orders of magnitude performance improvement in certain cases over standard regular expression matching systems, such as lex, awk and grep.
2425278
Shuffling a Stacked Deck: The Case for Partially Randomized Ranking of Search Engine Results
In-degree, PageRank, number of visits and other measures of Web page popularity significantly influence the ranking of search results by modern search engines. The assumption is that popularity is closely correlated with quality, a more elusive concept that is difficult to measure directly. Unfortunately, the correlation between popularity and quality is very weak for newly-created pages that have yet to receive many visits and/or in-links. Worse, since discovery of new content is largely done by querying search engines, and because users usually focus their attention on the top few results, newly-created but high-quality pages are effectively "shut out," and it can take a very long time before they become popular.We propose a simple and elegant solution to this problem: the introduction of a controlled amount of randomness into search result ranking methods. Doing so offers new pages a chance to prove their worth, although clearly using too much randomness will degrade result quality and annul any benefits achieved. Hence there is a tradeoff between exploration to estimate the quality of new pages and exploitation of pages already known to be of high quality. We study this tradeoff both analytically and via simulation, in the context of an economic objective function based on aggregate result quality amortized over time. We show that a modest amount of randomness leads to improved search results.
5814506
Efficient Monitoring Algorithm for Fast News Alerts
Recently, there has been a dramatic increase in the use of XML data to deliver information over the Web. Personal Weblogs, news Web sites, and discussion forums are now publishing RSS feeds for their subscribers to retrieve new postings. As the popularity of personal Weblogs and RSS feeds grows rapidly, RSS aggregation services and blog search engines have appeared, which try to provide a central access point for simpler access and discovery of new content from a large number of diverse RSS sources. In this paper, we study how the RSS aggregation services should monitor the data sources to retrieve new content quickly using minimal resources and to provide its subscribers with fast news alerts. We believe that the change characteristics of RSS sources and the general user access behavior pose distinct requirements that make this task significantly different from the traditional index refresh problem for Web search engines. Our studies on a collection of 10,000 RSS feeds reveal some general characteristics of the RSS feeds and show that, with proper resource allocation and scheduling, the RSS aggregator provides news alerts significantly faster than the best existing approach.
1176457
Effective Change Detection Using Sampling
For a large-scale data-intensive environment, such as the World-Wide Web or data warehousing, we often make local copies of remote data sources. Due to limited network and computational resources, however, it is often difficult to monitor the sources constantly to check for changes and to download changed data items to the copies. In this scenario, our goal is to detect as many changes as we can using the fixed download resources that we have. In this paper we propose three sampling-based download policies that can identify more changed data items effectively. In our sampling-based approach, we first sample a small number of data items from each data source and download more data items from the sources with more changed samples. We analyze the effectiveness of the sampling-based policies and compare our proposed policies to existing ones, including the state-of-the-art frequency-based policy in [8, 11]. Our experiments on synthetic and real-world data will show the relative merits of various policies and the great potential of our sampling-based policy. In certain cases, our sampling-based policy could download twice as many changed items as the best existing policy.
1720833
Automatic identification of user interest for personalized search
One hundred users, one hundred needs. As more and more topics are being discussed on the web and our vocabulary remains relatively stable, it is increasingly difficult to let the search engine know what we want. Coping with ambiguous queries has long been an important part of the research on Information Retrieval, but still remains a challenging task. Personalized search has recently got significant attention in addressing this challenge in the web search community, based on the premise that a user's general preference may help the search engine disambiguate the true intention of a query. However, studies have shown that users are reluctant to provide any explicit input on their personal preference. In this paper, we study how a search engine can learn a user's preference automatically based on her past click history and how it can use the user preference to personalize search results. Our experiments show that users' preferences can be learned accurately even from little click-history data and personalized search based on user preference yields significant improvements over the best existing ranking mechanism in the literature.
1534837
A Parallel Execution Model of Logic Program Based on Dependency Relationship Graph
6528301
Energy Efficient Broadcast in Wireless Ad hoc Networks with Hitch-hiking
In this paper, we have proposed a novel concept called Hitch-hiking in order to reduce the energy consumption of broadcast application for wireless networks. Hitch-hiking takes advantage of the physical layer design that facilitates the combining of partial signals to obtain the complete information. The concept of combining partial signals using maximal ratio combiner [13] has been used to improve the reliability of the communication channel but has never been exploited to reduce energy consumption. We propose a centralized heuristic algorithm called Wireless Multicast with Hitch-hiking (WMH) to construct an energy efficient tree using Hitch-hiking and also present a distributed version of the heuristic. We also evaluate the proposed heuristics through simulation. Simulation results show that Hitch-hiking can reduce the transmission cost of broadcast by as much as 50%. Further, we propose and evaluate a protocol called PSBT that reduces energy consumption of broadcast by eliminating redundancy in receive operation. Finally, we propose an algorithm that takes advantage of both Hitch-hiking and PSBT in conserving energy.
43684392
Wavelet Analysis Based Blind Watermarking for 3-D Surface Meshes
As most previous wavelet analysis based 3-D mesh watermarking methods embed the watermark information into wavelet coefficients arranged in a certain order, they have not been used as blind schemes since the connectivity information must be exactly known in the watermark extraction process. In this paper, we propose a blind watermarking method based on wavelet analysis for 3-D mesh model. Two new techniques are introduced. One is to exploit the statistical features of scale coefficients on an approximation (low resolution) level for watermark embedding. Another is to extract the hidden watermark, not from the same resolution level as used in embedding process, but directly from the spatial domain. As the proposed watermark detection does not require the wavelet analysis, any pre-processing such as registration and re-sampling, is not needed. These techniques allow to detect the watermark without referring to the original meshes. In addition, the proposed are applicable directly to irregular meshes by using irregular wavelet analysis. Through simulations, we prove that our method is fairly robust against various attacks including topological ones.
9811102
Four-bend top-down global routing
We propose a new global net distribution approach for high-performance m/spl times/m two-dimensional arrays of very large scale integration and multichip-modules. The objective is to route n nets with minimum density of global cells, using a "small" number of bends. There are a number of applications where it is necessary to limit the number of bends on each wire. For example, it is desirable to limit the number of bends on each microstrip (transmission) line, for mismatches of line impedance can cause reflections from the junction points such as bends and vias. Furthermore, for high-performance routing, intersections of wires cause the use of more vias, which in turn require the use of more routing resources (because of the larger via pitch). This is the first paper that addresses a graph-theoretic framework to solve the bend-constrained global routing problem in two-dimensional arrays. In this paper, at each level of an underlying quad-tree, we present a novel four-bend routing algorithm by decomposing the original problem at level i into two subproblems that can be solved exactly based on a two-stage approach of smaller-sized linear program followed by min-cost flow networks. The overall (i.e., entire level of the four-way partition hierarchy) constraint and variable size for the first stage is O(md/sub o/), while the overall run time for the second stage is O(n/sup 3/ log n/sup 2/). The time complexity of such a hierarchical approach is one order of magnitude less than one of constructing a global routing using the min-cost-flow-based flat design approach. Last, we present an extension that permits a limited degree of control over the number of bends. The proposed algorithm can also be used for estimating the wireability in the early design planning stage for high-level synthesis. Experimental results showed the effectiveness of the proposed algorithm.
1326236
History Sensitive String for Multiple Alphabets
In most cases, to represent characters in a computer, conventional code systems such as ASCII, EBCDIC, or BCD, whose code-length is uniform, have been used. In some countries, where the native language is not English, people can easily adapt their alphabet to these code systems by simply replacing the Enghsh alphabet with the alphabet of their native language. In other countries, especially in Korea and Japan, a straightforward application of the conventional code system is not possible. This is because, to compose a sentence, people customarily intermix words each of which may be constructed by different alphabets. For example, in Korea, they have an alphabet for their native language called Hangul. Hangul is a complete language by which Koreans can communicate with each other. However, Korean culture was influenced by China and more recently by the western countries. This results in widespread usage of four different types of documents in Korea. Documents may be written in Hangul only, Hangul with Chinese words, Hangul with English words, and Hangul with both Chinese and English words. Even in these cases, Hangul is the main alphabet. Another example of the applications of multiple alphabets in a single text are the high level language programs in Korean environment where the program language is a popular high level language such as PASCAL, C, or LISP, but variable names or data in the program may consist of Korean characters [7,9]. In this case, the main alphabet is English. Another example using heterogeneous characters is the use of attributes such as color, size, or font in high quality text editors. In such cases, characters from multiple alphabets should be used in a computer. Since each alphabet may have characteristics different from each other, some coding methods for multiple alphabets can be considered. When the sizes of the alphabets are significantly different from each other, employment of codes of a uniform length is prohibitively expensive. There are two different techniques to encode the multiple alphabets into a single code system. In the following sections, two coding methods of multiple alphabets are generalized, and their advantages and disadvantages are analyzed.