9081613
Stanford WebBase components and applications
We describe the design and performance of WebBase, a tool for Web research. The system includes a highly customizable crawler, a repository for collected Web pages, an indexer for both text and link-related page features, and a high-speed content distribution facility. The distribution module enables researchers world-wide to retrieve pages from WebBase, and stream them across the Internet at high speed. The advantage for the researchers is that they need not all crawl the Web before beginning their research. WebBase has been used by scores of research and teaching organizations world-wide, mostly for investigations into Web topology and linguistic content analysis. After describing the system's architecture, we explain our engineering decisions for each of the WebBase components, and present respective performance measurements.
1713822
A fast regular expression indexing engine
In this paper; we describe the design, architecture, and lessons learned from the implementation of a fast regular-expression indexing engine FREE. FREE uses a prebuilt index to identify the text data units which may contain a matching string and only examines these further. In this way, FREE shows orders of magnitude performance improvement in certain cases over standard regular expression matching systems, such as lex, awk and grep.
171503
Parallel crawlers
In this paper we study how we can design an effective parallel crawler. As the size of the Web grows, it becomes imperative to parallelize a crawling process, in order to finish downloading pages in a reasonable amount of time. We first propose multiple architectures for a parallel crawler and identify fundamental issues related to parallel crawling. Based on this understanding, we then propose metrics to evaluate a parallel crawler, and compare the proposed architectures using 40 million pages collected from the Web. Our results clarify the relative merits of each architecture and provide a good guideline on when to adopt which architecture.
2425278
Shuffling a Stacked Deck: The Case for Partially Randomized Ranking of Search Engine Results
In-degree, PageRank, number of visits and other measures of Web page popularity significantly influence the ranking of search results by modern search engines. The assumption is that popularity is closely correlated with quality, a more elusive concept that is difficult to measure directly. Unfortunately, the correlation between popularity and quality is very weak for newly-created pages that have yet to receive many visits and/or in-links. Worse, since discovery of new content is largely done by querying search engines, and because users usually focus their attention on the top few results, newly-created but high-quality pages are effectively "shut out," and it can take a very long time before they become popular.We propose a simple and elegant solution to this problem: the introduction of a controlled amount of randomness into search result ranking methods. Doing so offers new pages a chance to prove their worth, although clearly using too much randomness will degrade result quality and annul any benefits achieved. Hence there is a tradeoff between exploration to estimate the quality of new pages and exploitation of pages already known to be of high quality. We study this tradeoff both analytically and via simulation, in the context of an economic objective function based on aggregate result quality amortized over time. We show that a modest amount of randomness leads to improved search results.
516682
Efficient Crawling Through URL Ordering
In this paper we study in what order a crawler should visit the URLs it has seen, in order to obtain more "important" pages first. Obtaining important pages rapidly can be very useful when a crawler cannot visit the entire Web in a reasonable amount of time. We define several importance metrics, ordering schemes, and performance evaluation measures for this problem. We also experimentally evaluate the ordering schemes on the Stanford University Web. Our results show that a crawler with a good ordering scheme can obtain important pages significantly faster than one without.
1739520
Exploiting Geographical Location Information of Web Pages
Many information resources on the web are relevant primarily to limited geographical communities. For instance, web sites containing information on restaurants, theaters, and apartment rentals are relevant primarily to web users in geographical proximity to these locations. In contrast, other information resources are relevant to a broader geographical community. For instance, an on-line newspaper may be relevant to users across the United States. Unfortunately, the geographical scope of web resources is largely ignored by web search engines. We make the case for identifying and exploiting the geographical location information of web sites so that web search engines can rank resources in a geographically sensitive fashion, in addition to using more traditional information-retrieval strategies. In this paper, we first consider how to compute the geographical location of web pages. Subsequently, we consider how to exploit such information in one specific "proof-of-concept" application we implemented in JAVA, and discuss other examples as well.
1326236
History Sensitive String for Multiple Alphabets
In most cases, to represent characters in a computer, conventional code systems such as ASCII, EBCDIC, or BCD, whose code-length is uniform, have been used. In some countries, where the native language is not English, people can easily adapt their alphabet to these code systems by simply replacing the Enghsh alphabet with the alphabet of their native language. In other countries, especially in Korea and Japan, a straightforward application of the conventional code system is not possible. This is because, to compose a sentence, people customarily intermix words each of which may be constructed by different alphabets. For example, in Korea, they have an alphabet for their native language called Hangul. Hangul is a complete language by which Koreans can communicate with each other. However, Korean culture was influenced by China and more recently by the western countries. This results in widespread usage of four different types of documents in Korea. Documents may be written in Hangul only, Hangul with Chinese words, Hangul with English words, and Hangul with both Chinese and English words. Even in these cases, Hangul is the main alphabet. Another example of the applications of multiple alphabets in a single text are the high level language programs in Korean environment where the program language is a popular high level language such as PASCAL, C, or LISP, but variable names or data in the program may consist of Korean characters [7,9]. In this case, the main alphabet is English. Another example using heterogeneous characters is the use of attributes such as color, size, or font in high quality text editors. In such cases, characters from multiple alphabets should be used in a computer. Since each alphabet may have characteristics different from each other, some coding methods for multiple alphabets can be considered. When the sizes of the alphabets are significantly different from each other, employment of codes of a uniform length is prohibitively expensive. There are two different techniques to encode the multiple alphabets into a single code system. In the following sections, two coding methods of multiple alphabets are generalized, and their advantages and disadvantages are analyzed.
7663373
Performance analysis of multipath multistage interconnection networks with nonuniform output traffic distribution
Develops the analysis of output buffers in multipath multistage interconnection networks. The delay-throughput performance of output buffers in multipath interconnection networks is closely related to the output traffic distribution, which is the distribution of packet arrival rates at each output link connected to a given output module. Many multipath networks produce nonuniform output traffic distributions, even if the input traffic pattern is uniform. We propose in this paper a performance model for output buffer analysis with respect to such a nonuniform output traffic distribution and analyze the output buffer performance of several multipath networks by our proposed analysis model. It is shown that the output traffic distributions are different for the various multipath networks and the output buffer performance (such as packet loss probability and delay) gets better as the nonuniformity of the output traffic distribution becomes higher.
9493395
3-D Dynamic Mesh Compression using Wavelet-Based Multiresolution Analysis
In this paper, we present a wavelet-based progressive compression method for 3-D dynamic meshes. Our method exploits the spatial and temporal redundancy. We encode the geometry of base mesh, the wavelet coefficients and the connectivity of each resolution level in order to reduce the spatial redundancy of intra meshes. For inter mesh coding, we encode the differences of geometry of base meshes and of their wavelet coefficients between adjacent frames to reduce the temporal redundancy. Our proposal is based on the wavelet-based multiresolution analysis which uses a perfect reconstruction filter bank and therefore it enables not only progressive representation but also lossless compression. The simulation results demonstrate that the proposed method is applicable to lossy and lossless compression of 3-D dynamic meshes.
46360239
A New Algorithm for Subsumption Test
19746677
Performance analysis of an ATM switch with multiple paths
An ATM switch based on multistage interconnection networks with multiple paths can support higher bandwidth than that of buffered networks with single path by passing multiple packets to the same destination simultaneously. These multiple packets are buffered in the output buffer of the destination. The performance of output buffer in the switch with multiple paths is closely dependent on the output traffic distribution, which is the packet arrival rate at each output link destined to a given output port. As the nonuniformity of the output traffic distribution becomes higher, the performance of the output buffer as like delay and packet loss probability gets better. In this paper, we propose a new self-routing switch architecture with multiple paths called Fly network and analyze the performance of the switching network and the output buffer focusing on the output traffic distribution. It is shown that the Fly network supports best throughput and latency of the output buffer by producing a high degree of the nonuniform output traffic distribution.