2659
Toward a unified approach to statistical language modeling for Chinese
This article presents a unified approach to Chinese statistical language modeling (SLM). Applying SLM techniques like trigram language models to Chinese is challenging because (1) there is no standard definition of words in Chinese; (2) word boundaries are not marked by spaces; and (3) there is a dearth of training data. Our unified approach automatically and consistently gathers a high-quality training data set from the Web, creates a high-quality lexicon, segments the training data using this lexicon, and compresses the language model, all by using the maximum likelihood principle, which is consistent with trigram model training. We show that each of the methods leads to improvements over standard SLM, and that the combined method yields the best pinyin conversion result reported.
8204298
Improved Source-Channel Models for Chinese Word Segmentation
This paper presents a Chinese word segmentation system that uses improved source-channel models of Chinese sentence generation. Chinese words are defined as one of the following four types: lexicon words, morphologically derived words, factoids, and named entities. Our system provides a unified approach to the four fundamental features of word-level Chinese language processing: (1) word segmentation, (2) morphological analysis, (3) factoid detection, and (4) named entity recognition. The performance of the system is evaluated on a manually annotated test set, and is also compared with several state-of-the-art systems, taking into account the fact that the definition of Chinese words often varies from system to system.
12419149
On the use of words and n-grams for Chinese information retrieval
In the processing of Chinese documents and queries in information retrieval (IR), one has to identify the units that are used as indexes. Words and n-grams have been used as indexes in several previous studies, which showed that both kinds of indexes lead to comparable IR performances. In this study, we carry out more experiments on different ways to segment documents and queries, and to combine words with n-grams. Our experiments show that a combination of the longest-matching algorithm with single characters is the best choice.
11423197
Unsupervised Learning of Dependency Structure for Language Modeling
This paper presents a dependency language model (DLM) that captures linguistic constraints via a dependency structure, i.e., a set of probabilistic dependencies that express the relations between headwords of each phrase in a sentence by an acyclic, planar, undirected graph. Our contributions are three-fold. First, we incorporate the dependency structure into an n-gram language model to capture long distance word dependency. Second, we present an unsupervised learning method that discovers the dependency structure of a sentence using a bootstrapping procedure. Finally, we evaluate the proposed models on a realistic application (Japanese Kana-Kanji conversion). Experiments show that the best DLM achieves an 11.3% error rate reduction over the word trigram model.
9663679
The Use of SVM for Chinese New Word Identification
We present a study of new word identification (NWI) to improve the performance of a Chinese word segmenter. In this paper the distribution and types of new words are discussed empirically. In particular, we focus on the new words of two surface patterns, which account for more than 80% of new words in our data sets: NW11 (two-character new word) and NW21 (a bi-character word followed with a single character). NWI is defined as a problem of binary classification. A statistical learning approach based on a SVM classifier is used. Different features for NWI are explored, including in-word probability of a character (IWP), the analogy between new words and lexicon words, anti-word list, and frequency in documents. The experiments show that these features are useful for NWI. The F-scores of NWI we achieved are 64.4% and 54.7% for NW11 and NW21, respectively. The overall performance of the Chinese word segmenter could be improved by Roov 24.5% and F-score 6.5% in PK-close test of the 1st SIGHAN bakeoff. This achieves the performance of state-of-the-art word segmenters.
461312
Dependence language model for information retrieval
This paper presents a new dependence language modeling approach to information retrieval. The approach extends the basic language modeling approach based on unigram by relaxing the independence assumption. We integrate the linkage of a query as a hidden variable, which expresses the term dependencies within the query as an acyclic, planar, undirected graph. We then assume that a query is generated from a document in two stages: the linkage is generated first, and then each term is generated in turn depending on other related terms according to the linkage. We also present a smoothing method for model parameter estimation and an approach to learning the linkage of a sentence in an unsupervised manner. The new approach is compared to the classical probabilistic retrieval model and the previously proposed language models with and without taking into account term dependencies. Results show that our model achieves substantial and significant improvements on TREC collections.
40124869
Image Pattern Recognition Based on Examples - A Combined Statistical and Structural-Syntactic Approach
An application of combined statistical and structural-syntactic approach in Chinese character recognition is presented. The algorithm adopts a structural representation for Chinese characters, but in the classification and training process, the structural matching and parameter adjustment is conducted in a statistical way. Different from the conventional structural approaches, in this system, only a few predefined "knowledge" is required. In most cases, knowledge acquisition is simplified to "memorization" of examples, and the parameters for classification can be refined using statistical training. In this way it avoids the main difficulties inherent in the implementation of classification systems based on structural features. Compared with conventional statistical algorithms, the algorithm is based on a structural model of image patterns, so it has approximately all the advantages of structural pattern recognition algorithms. A prototype system has been realized based on this strategy, and the effectiveness of the method is verified by the experimental results.
3057656
Scalable Data Servers for Large Multivariate Volume Visualization
Volumetric datasets with multiple variables on each voxel over multiple time steps are often complex, especially when considering the exponentially large attribute space formed by the variables in combination with the spatial and temporal dimensions. It is intuitive, practical, and thus often desirable, to interactively select a subset of the data from within that high-dimensional value space for efficient visualization. This approach is straightforward to implement if the dataset is small enough to be stored entirely in-core. However, to handle datasets sized at hundreds of gigabytes and beyond, this simplistic approach becomes infeasible and thus, more sophisticated solutions are needed. In this work, we developed a system that supports efficient visualization of an arbitrary subset, selected by range-queries, of a large multivariate time-varying dataset. By employing specialized data structures and schemes of data distribution, our system can leverage a large number of networked computers as parallel data servers, and guarantees a near optimal load-balance. We demonstrate our system of scalable data servers using two large time-varying simulation datasets
6460806
Distributed Proximity Maintenance in Ad Hoc Mobile Networks
We present an efficient distributed data structure, called the d-Spanner, for maintaining proximity information among communicating mobile nodes. The d-Spanner is a kinetic sparse graph spanner on the nodes that allows each node to quickly determine which other nodes are within a given distance of itself, to estimate an approximate nearest neighbor, and to perform a variety of other proximity related tasks. A lightweight and fully distributed implementation is possible, in that maintenance of the proximity information only requires each node to exchange a modest number of messages with a small number of mostly neighboring nodes. The structure is based on distance information between communicating nodes that can be derived using ranging or localization methods and requires no additional shared infrastructure other than an underlying communication network. Its modest requirements make it scalable to networks with large numbers of nodes.
14630821
A New Semi-Supervised Subspace Clustering Algorithm on Fitting Mixture Models
We propose a new subspace clustering algorithm (EPSCMIX), which is based on the feature saliency measure that is obtained by using both the Emerging Patterns algorithm and the EM algorithm, for the analysis of microarray data. For the model selection, it employs a novel agglomerative step together with MDL criterion. And, we present the result of comparative experiments between AIC, MDL and minimum message length (MML) used to determine a criterion for our algorithm. The robustness of using emerging patterns based on mixture models, as well as using the Gaussian mixture model for subspace clustering, was demonstrated on both synthetic and real data sets. In experiments, it also certified that a new agglomerative method that merges mostly correlated components with MDL consistently worked better than the one that removes weak weight components.
15355925
Customizable virtual private network service with QoS
Abstract In this paper, we propose and implement Virtual Network Service (VNS), a value-added network service for deploying virtual private networks (VPNs) in a managed wide-area IP network. The key feature of VNS is its capability of providing a customer with a VPN that is customizable with management capabilities and performance properties comparable to a dedicated physical network. In addition, VNS ensures confidentiality of data and principals through the use of IPSEC. The main technique underlying VNS is the virtualization of routers in both control and data planes. Virtualization of the control plane enables customizable routing and signaling per VPN. On the data plane, packet forwarding and link bandwidth are virtualized. Virtualization of the forwarding mechanism on the data plane enables forwarding of traffic according to each VPN's topology and policies. Virtualization of the link bandwidth enables each VPN to have guaranteed quality of service (QoS) and customized resource management policies. We have developed a VNS prototype for deployment on the CAIRN network. The VNS prototype implements several resource management mechanisms including packet scheduling, signaling and runtime monitoring. A graphical user interface enables service providers to manage, configure and deploy VPNs remotely.