206718678
A General Convergence Theorem for the Decomposition Method
The decomposition method is currently one of the major methods for solving the convex quadratic optimization problems being associated with support vector machines. Although there exist some versions of the method that are known to converge to an optimal solution, the general convergence properties of the method are not yet fully understood. In this paper, we present a variant of the decomposition method that basically converges for any convex quadratic optimization problem provided that the policy for working set selection satisfies three abstract conditions. We furthermore design a concrete policy that meets these requirements.
1182152
Stability of k -Means Clustering
We consider the stability of k-means clustering problems. Clustering stability is a common heuristics used to determine the number of clusters in a wide variety of clustering applications. We continue the theoretical analysis of clustering stability by establishing a complete characterization of clustering stability in terms of the number of optimal solutions to the clustering optimization problem. Our results complement earlier work of Ben-David, von Luxburg and Pal, by settling the main problem left open there. Our analysis shows that, for probability distributions with finite support, the stability of k-means clusterings depends solely on the number of optimal solutions to the underlying optimization problem for the data distribution. These results challenge the common belief and practice that view stability as an indicator of the validity, or meaningfulness, of the choice of a clustering algorithm and number of clusters.
45719384
A Characterization of Strong Learnability in the Statistical Query Model
In this paper, we consider Kearns' [4] Statistical Query Model of learning. It is well known [3] that the number of statistical queries, needed for "weakly learning" an unknown target concept (i.e. for gaining significant advantage over random guessing) is polynomially related to the so-called Statistical Query dimension of the concept class. In this paper, we provide a similar characterization for "strong learning" where the learners final hypothesis is required to approximate the unknown target concept up to a small rate of misclassification. The quantity that characterizes strong learnability in the Statistical Query model is a surprisingly close relative of (though not identical to) the Statistical Query dimension. For the purpose of proving the main result, we provide other characterizations of strong learnability which are given in terms of covering numbers and related notions. These results might find some interest in their own right. All characterizations are purely information-theoretical and ignore computational issues.
7797628
Bayesian Networks and Inner Product Spaces
In connection with two-label classification tasks over the Boolean domain, we consider the possibility to combine the key advantages of Bayesian networks and of kernel-based learning systems. This leads us to the basic question whether the class of decision functions induced by a given Bayesian network can be represented within a low-dimensional inner product space. For Bayesian networks with an explicitly given (full or reduced) parameter collection, we show that the “natural” inner product space has the smallest possible dimension up to factor 2 (even up to an additive term 1 in many cases). For a slight modification of the so-called logistic autoregressive Bayesian network with n nodes, we show that every sufficiently expressive inner product space has dimension at least 2 n/4. The main technical contribution of our work consists in uncovering combinatorial and algebraic structures within Bayesian networks such that known techniques for proving lower bounds on the dimension of inner product spaces can be brought into play.
31930258
How Many Missing Answers Can Be Tolerated by Query Learners?
We consider the model of exact learning using an equivalence oracle and an incomplete membership oracle. In this model, a random subset of the learners membership queries is left unanswered. Our results are as follows. First, we analyze the obvious method for coping with missing answers: search exhaustively through all possible "answer patterns" associated with the unanswered queries. Thereafter, we present two specific concept classes that are efficiently learnable using an equivalence oracle and a (completely reliable) membership oracle, but are provably not polynomially learnable if the membership oracle becomes slightly incomplete. The first class will demonstrate that the aforementioned method of exhaustively searching through all possible answer patterns cannot be substantially improved in general (despite its apparent simplicity). The second class will demonstrate that the incomplete membership oracle can be rendered useless even if it leaves only a fraction 1/poly(n) of all queries unanswered. Finally, we present a learning algorithm for monotone DNF formulas that can cope with a relatively large fraction of missing answers (more than sixty percent), but is as efficient (in terms of run-time and number of queries) as the classical algorithm whose questions are always answered reliably.
37804897
Directions for Qualitative Reasoning
Sacks & Doyle provide an excellent overview of the fundamental limitations of the SPQR representations for reasoning about the qualitative properties of dynamic systems. We take this opportunity to outline some new directions for qualitative reasoning. In this paper, we provide a rigorous mathematical characterization for the term “qualitative property” in the context of static and dynamic systems. Based on these characterizations, we show that interval representations are well suited for reasoning about the qualitative properties of static systems such as qualitative comparative statics and qualitative stability. Moreover, we also show that symbolic computations help in the derivation of useful global properties of dynamic systems which can be used to guide numerical sampling of differential equations. The integration of symbolic and numeric methods provides a powerful approach for automating the qualitative analysis of differential equations.
6270234
Spectral Relaxation for K-means Clustering
The popular K-means clustering partitions a data set by minimizing a sum-of-squares cost function. A coordinate descend method is then used to find local minima. In this paper we show that the minimization can be reformulated as a trace maximization problem associated with the Gram matrix of the data vectors. Furthermore, we show that a relaxed version of the trace maximization problem possesses global optimal solutions which can be obtained by computing a partial eigendecomposition of the Gram matrix, and the cluster assignment for each data vectors can be found by computing a pivoted QR decomposition of the eigenvector matrix. As a by-product we also derive a lower bound for the minimum of the sum-of-squares cost function.
29911339
Artificial Intelligence: An Empirical Science
My initial tasks in this paper are, first, to delimit the boundaries of artificial intelligence, then, to justify calling it a science: is AI science, or is it engineering, or some combination of these? After arguing that it is (at least) a science, I will consider how it is best pursued: in particular, the respective roles for experiment and theory in developing AI. I will rely more on history than on speculation, for our actual experience in advancing the field has much to tell us about how we can continue and accelerate that advance. Many of my examples will be drawn from work with which I have been associated, for I can speak with greater confidence about what motivated that work and its methods (and about its defects) than I can about the work of others. My goal, however, is not to give you a trip through history, but to make definite proposals for our future priorities, using history, where relevant, as evidence for my views.
15281282
Fast multilevel implementation of recursive spectral bisection for partitioning unstructured problems
SUMMARY If problems involving unstructured meshes are to be solved efficiently on distributed-memory parallel computers, the meshes must be partitioned and distributed across processors in a way that balances the computational load and minimizes communication. The recursive spectral bisection method (RSB) has been shown to be very effective for such partitioning problems compared to alternative methods, but RSB in its simplest form is expensive. Here a multilevel version of RSB is introduced that attains about an order-of-magnitude improvement in run time on typical examples. 1. INTRODUCTION Unstructured meshes are used in several large-scale scientific and engineering problems, including finite-volume methods for computational fluid dynamics and finite-element methods for structural analysis. If unstructured problems such as these are to be solved on distributed-memory parallel computers, their data structures must be partitioned and distributed across processors; if they are to be solved efficiently, the partitioning must niaximize load balance and minimize interprocessor communication. Recently, the recursive spectral bisection method (RSB)[l] has been shown to be very effective for such partitioning problems compared to alternative methods. Unfortunately, RSB in its simplest form is expensive. We shall describe a multilevel version of RSB that attains about im order-of-magnitude improvement in run time on typical examples.
35264400
Parallel Efficiency of the Lanczos Method for Eigenvalue Problems
LBNL-42828 Parallel Efficiency of the Lanczos Method for Eigenvalue Problems ^ Kesheng Wu+and Horst Simon* Abstract Two of the commonly used versions of the Lanczos method for eigenvalues problems are the shift-and-invert Lanczos method and the restarted Lanczos method. In this talk, we will address two questions, is the shift-and-invert Lanczos method a viable option on massively parallel machines and which one is more appropriate for a given eigenvalue problem? Introduction T h i s talk is o n how to compute eigenvalues a n d eigenvectors of large sparse s y m m e t r i c matrices o n massively parallel machines. One of the the most c o m m o n l y used algorithms for this task is the Lanczos m e t h o d w h i c h projects the large eigenvalue p r o b l e m onto a low dimensional K r y l o v subspace [7, 10]. I n most cases, the K r y l o v subspace basis is built t h r o u g h a series of matrix-vector m u l t i p l i c a t i o n s , the whole Lanczos m e t h o d c a n be implemented w i t h a matrix-vector m u l t i p l i c a t i o n routine a n d a few simple vector operations. T h i s a l g o r i t h m is highly efficient on parallel machines a n d it is effective for c o m p u t i n g extreme a n d well separated eigenvalues. To compute the interior or not well-separated eigenvalues, the shift-and-invert Lanczos m e t h o d is one of the most effective methods. Since the eigenvalues of a m a t r i x A are related to the eigenvalues of (A — c r l ) by a simple relation X(A) = a + 1/X((A — c r l ) ) a n d the corresponding eigenvectors of A a n d (A — c r l ) are identical, the shift-and-invert scheme computes the extreme eigenvalues of (A — c r l ) a n d deduce the corresponding eigenvalues of A. W i t h appropriate choice of a, the extreme eigenvalues of (A — al)^ are well separated and can be easily computed. T h e shift-and-invert Lanczos m e t h o d needs to b u i l d a K r y l o v subspace basis of (A — c r l ) w h i c h is done by solving a series of linear systems involving the coefficient m a t r i x (A — a I). T h e linear systems need to be solved accurately a n d the only reliable means to accomplish this is by using a direct m e t h o d [5]. Because it is difficult to implement direct methods o n d i s t r i b u t e d parallel computers, efficient parallel i m p l e m e n t a t i o n has not been widely available u n t i l recently [1, 8]. T h i s talk w i l l present our s t u d y of the parallel efficiency of the shift-and-invert Lanczos m e t h o d using these newly available direct solvers. T h e study w i l l show that the shift-and-invert Lanczos m e t h o d is ' T h i s w o r k was s u p p o r t e d b y t h e D i r e c t o r , Office o f E n e r g y R e s e a r c h , Office o f L a b o r a t o r y P o l i c y a n d Infrastructure M a n a g e m e n t , of the U . S . D e p a r t m e n t of E n e r g y under C o n t r a c t N o . D E - A C 0 3 - 7 6 S F 0 0 0 9 8 . T h i s r e s e a r c h u s e d r e s o u r c e s o f t h e N a t i o n a l E n e r g y R e s e a r c h S c i e n t i f i c C o m p u t i n g C e n t e r , w h i c h is s u p p o r t e d b y t h e Office o f E n e r g y R e s e a r c h o f t h e U . S . D e p a r t m e n t o f E n e r g y . 'Lawrence Berkeley National Laboratory/NERSC, Berkeley, CA Email: {ksni, hdsimon}81bl.gov.