11235220
Probability-Based Chinese Text Processing and Retrieval
We discuss the use of probability-based natural language processing for Chinese text retrieval. We focus on comparing different text extraction methods and probabilistic weighting methods. Several docu- ment processing methods and probabilistic weighting functions are presented. A number of experiments have been conducted on large standard text collections. We present the experimental results that com- pare a word-based text processing method with a character-based method. The experimental results also compare a number of term-weighting functions including both single-unit weighting and compound-unit weighting functions. Modern information retrieval (IR) systems should be able to take natural language queries and retrieve documents written in natural languages. One way to process nat- ural language texts in IR is statistical. In this approach, linguistic units are extracted from the documents and queries. Those units extracted from the documents are used to index the documents. Those extracted from a query are weighted according to their degrees of importance. The most important units are chosen as query terms. An infor- mation retrieval system takes the selected query terms, matches the terms with the indexes of documents, calculates a retrieval status value for each matched document using a term-weighting method, and presents the user with potentially relevant docu- ments from a collection of documents. The performance of the information retrieval system in identifying relevant documents greatly depends on the text processing method used. We investigate the effectiveness of different text extraction methods and different term weighting methods in the context of Chinese information retrieval. It is a well- known problem that there is no separator between Chinese words, so Chinese words cannot be used easily to index or search texts as is possible in English. Therefore, some people use characters or n-grams as searchable tokens instead of words. We discuss two text extraction methods. One extraction method uses words, compound words, and phrases in the document and query texts as indexing terms to represent the texts. We refer to this method as the word-based approach. For this approach, text segmentation, which divides both document and query texts into linguistic units, is regarded not only as a necessary precursor but also as a bottleneck of this kind of system (Wu and Tseng 1993). The other extraction method is based on single Chinese characters, in which texts are indexed by the characters appearing in the texts (Chen 1992). By using sin- gle character approaches, a search could be conducted for any multi-character word or phrase identified at search time, no matter whether this word or phrase is in the dic- tionary. Both word-based and character-based methods have been used in information
14449557
Evaluation and automatic selection of methods for handling missing data
Real-world data often contain missing values. In this paper, nine different methods for handling missing data are described and compared. Thirty real-world datasets are used as input data in our experiment. Nine different missing levels are implemented in each training data set to investigate the performance of the methods at different missing levels. For evaluation, two different classification techniques, C4.5 and ELEM2, are applied to obtain the classification error rates. The experimental results show that different methods can have different effects on a data set. Based on the experimental results, we also propose a meta-learning approach to selecting a method for handling missing data. The evaluation results show that the meta-learning approach is effective in selecting a suitable method for handling missing data.
921552
Dynamic Web log session identification with statistical language models
We present a novel session identification method based on statistical language modeling. Unlike standard time-out methods, which use fixed time thresholds for session identification, we use an information theoretic approach that yields more robust results for identifying session boundaries. We evaluate our new approach by learning interesting association rules from the segmented session files. We then compare the performance of our approach to three standard session identification method--the standard timeout method, the reference length method, and the maximal forward reference method--and find that our statistical language modeling approach generally yields superior results. However, as with every method, the performance of our technique varies with changing parameter settings. Therefore, we also analyze the influence of the two key factors in our language-modeling-based approach: the choice of smoothing technique and the language model order. We find that all standard smoothing techniques, save one, perform well, and that performance is robust to language model order.
27390518
Session Boundary Detection for Association Rule Learning Using n-Gram Language Models
We present a statistical method using n-gram language models to identify session boundaries in a large collection of Livelink log data. The identified sessions are then used for association rule learning. Unlike the traditional ad hoc timeout method, which uses fixed time thresholds for session identification, our method uses an information theoretic approach that provides a natural technique for performing dynamic session identification. The effectiveness of our approach is evaluated with respect to 4 different interestingness measures. We find that we obtain a significant improvement in each interestingness measure, ranging from a 26.6% to 39% improvement on average over the best results obtained with standard timeout methods.
668000
Applying language modeling to session identification from database trace logs
A database session is a sequence of requests presented to the database system by a user or an application to achieve a certain task. Session identification is an important step in discovering useful patterns from database trace logs. The discovered patterns can be used to improve the performance of database systems by prefetching predicted queries, rewriting the current query or conducting effective cache replacement.In this paper, we present an application of a new session identification method based on statistical language modeling to database trace logs. Several problems of the language modeling based method are revealed in the application, which include how to select values for the parameters of the language model, how to evaluate the accuracy of the session identification result and how to learn a language model without well-labeled training data. All of these issues are important in the successful application of the language modeling based method for session identification. We propose solutions to these open issues. In particular, new methods for determining an entropy threshold and the order of the language model are proposed. New performance measures are presented to better evaluate the accuracy of the identified sessions. Furthermore, three types of learning methods, namely, learning from labeled data, learning from semi-labeled data and learning from unlabeled data, are introduced to learn language models from different types of training data. Finally, we report experimental results that show the effectiveness of the language model based method for identifying sessions from the trace logs of an OLTP database application and the TPC-C Benchmark.
9414412
Concept-based biomedical text retrieval
One challenging problem for biomedical text retrieval is to find accurate synonyms or name variants for biomedical entities. In this paper, we propose a new concept-based approach to tackle this problem. In this approach, a set of concepts instead of keywords will be extracted from a query first. Then these concepts will be used for retrieval purpose. The experiment results show that the proposed approach can boost the retrieval performance and it generates very good results on 2005 TREC Genomics data sets.
18862916
Clustering graphs for visualization via node similarities
Graph visualization is commonly used to visually model relations in many areas. Examples include Web sites, CASE tools, and knowledge representation. When the amount of information in these graphs becomes too large, users, however, cannot perceive all elements at the same time. A clustered graph can greatly reduce visual complexity by temporarily replacing a set of nodes in clusters with abstract nodes. This paper proposes a new approach to clustering graphs. The approach constructs the node similarity matrix of a graph that is derived from a novel metric of node similarity. The linkage pattern of the graph is thus encoded into the similarity matrix, and then one obtains the hierarchical abstraction of densely linked subgraphs by applying the k-means algorithm to the matrix. A heuristic method is developed to overcome the inherent drawbacks of the k-means algorithm. For clustered graphs we present a multilevel multi-window approach to hierarchically drawing them in different abstract level views with the purpose of improving their readability. The proposed approaches demonstrate good results in our experiments. As application examples, visualization of part of Java class diagrams and Web graphs are provided. We also conducted usability experiments on our algorithm and approach. The results have shown that the hierarchically clustered graph used in our system can improve user performance for certain types of tasks.
42013296
On the computational hardness based on linear FPT-reductions
The notion of linear fpt-reductions has been recently introduced to derive strong computational lower bounds for well-known NP-hard problems. In this paper, we formally investigate the notion of W[t]-hardness under the linear fpt-reduction, and study the structural properties of the corresponding complexity classes. Additional complexity lower bounds on important computational problems are established. Some observations on structural properties of the standard parameterized hierarchy, the W -hierarchy, are also presented.
39883279
Identifying Clusters in the Web Graph by Using Local Search
33920606
Hot-Spot Passage Retrieval in Question Answering
Question Answering has been the recent focus of information retrieval research; many systems just incorporate a search engine as a black box and most effort has been devoted to the question analyzer and the answer identifier. In the context of QA, however, passage provides an ideal medium between the document collection and an exact answer. And passage retrieval is a finer-grain approach than the traditional document retrieval both for the answer identifier and a human reader. In this paper, distinctions are first made between document retrieval and passage retrieval. And the Hot-Spot Passage Retrieval algorithm, which takes into account the measures of blurred BM25, coverage and height, is examined in detail. For evaluation, an isolated test is conducted and the algorithm gains 18.3% better answer redundancy and 4.8% better coverage rate than Okapi's original passage retrieval algorithm.
12321641
Locally Balanced Incremental Hierarchical Discriminant Regression
Incremental hierarchical discriminant regression faces several challenging issues: (a) a large input space with a small output space and (b) nonstationary statistics of data sequences. In the first case (a), there maybe few distinct labels in the output space while the input data distribute in a high dimensional space. In the second case (b), a tree has to be grown when only a limited data sequence has been observed. In this paper, we present the Locally Balanced Incremental Hierarchical Discriminant Regression (LBIHDR) algorithm. A novel node self-organization and spawning strategy is proposed to generate a more discriminant subspace by forming multiple clusters for one class. The algorithm was successfully applied to different kinds of data set.