14688234
Relational Knowledge with Predictive State Representations
Most work on Predictive Representations of State (PSRs) has focused on learning and planning in unstructured domains (for example, those represented by flat POMDPs). This paper extends PSRs to represent relational knowledge about domains, so that they can use policies that generalize across different tasks, capture knowledge that ignores irrelevant attributes of objects, and represent policies in a way that is independent of the size of the state space. Using a blocks world domain, we show how generalized predictions about the future can compactly capture relations between objects, which in turn can be used to naturally specify relational-style options and policies. Because our representation is expressed solely in terms of actions and observations, it has extensive semantics which are statistics about observable quantities.
2624439
Learning Predictive State Representations
We introduce the first algorithm for learning predictive state representations (PSRs), which are a way of representing the state of a controlled dynamical system. The state representation in a PSR is a vector of predictions of tests, where tests are sequences of actions and observations said to be true if and only if all the observations occur given that all the actions are taken. The problem of finding a good PSR. one that is a sufficient statistic for the dynamical system--can be divided into two parts: 1) discovery of a good set of tests, and 2) learning to make accurate predictions for those tests. In this paper, we present detailed empirical results using a gradient-based algorithm for addressing the second problem. Our results demonstrate several sample systems in which the algorithm learns to make correct predictions and several situations in which the algorithm is less successful. Our analysis reveals challenges that will need to be addressed in future PSR learning algorithms.
2695116
Near-Optimal Reinforcement Learning in Polynomial Time
We present new algorithms for reinforcement learning and prove that they have polynomial bounds on the resources required to achieve near-optimal return in general Markov decision processes. After observing that the number of actions required to approach the optimal return is lower bounded by the mixing time T of the optimal policy (in the undiscounted case) or by the horizon time T (in the discounted case), we then give algorithms requiring a number of actions and total computation time that are only polynomial in T and the number of states and actions, for both the undiscounted and discounted cases. An interesting aspect of our algorithms is their explicit handling of the Exploration-Exploitation trade-off.
2661414
Graphical Models for Game Theory
We introduce a compact graph-theoretic representation for multi-party game theory. Our main result is a provably correct and efficient algorithm for computing approximate Nash equilibria in one-stage games represented by trees or sparse graphs.
5053575
Bias-Variance Error Bounds for Temporal Difference Updates
Temporal difference (td) algorithms are used in reinforcement learning to compute estimates of the value of a given policy in an unknown Markov decision process (policy evaluation). We give rigorous upper bounds on the error of the closely related phased td algorithms (which differ from the standard updates in their treatment of the learning rate) as a function of the amount of experience. These upper bounds prove exponentially fast convergence, with both the rate of convergence and the asymptote strongly dependent on the length of the backups k or the parameter . Our bounds give formal verification to the well-known intuition that td methods are subject to a bias-variance tradeoff, and they lead to schedules for k and that are predicted to be better than any fixed values for these parameters. We give preliminary experimental confirmation of our theory for a version of the random walk problem.
7241207
Intrinsically Motivated Reinforcement Learning
Psychologists call behavior intrinsically motivated when it is engaged in for its own sake rather than as a step toward solving a specific problem of clear practical value. But what we learn during intrinsically motivated behavior is essential for our development as competent autonomous entities able to efficiently solve a wide range of practical problems as they arise. In this paper we present initial results from a computational study of intrinsically motivated reinforcement learning aimed at allowing artificial agents to construct and extend hierarchies of reusable skills that are needed for competent autonomy.
570980
An efficient on-line path planner for outdoor mobile robots
Abstract Mobile robots operating in outdoor unstructured environments often have only incomplete maps and must deal with new objects found during traversal. Path planning in these environments must be incremental to accommodate new information and must use efficient representations. This article reports recent results in path planning using an efficient data structure (framed-quadtrees) and an optimal algorithm (D*) to incrementally replan optimal paths. We show how the use of framed-quadtrees leads to paths that are shorter and more direct than when other representations are used. We also show the difference in performance when the robot starts with no information about the world versus when it starts with partial information about the world. Our results indicate that, as would be expected, starting with partial information is better than starting with no information. However, in many cases, partial information results in performance that is almost as good as starting out with complete information about the world, while the computational cost incurred is significantly lower. Our system has been tested in simulation as well on an autonomous jeep equipped with local obstacle avoidance capabilities. Results from both simulation and real experimentation are discussed.
2924964
Exploiting heterogeneity in sensor networks
The presence of heterogeneous nodes (i.e., nodes with an enhanced energy capacity or communication capability) in a sensor network is known to increase network reliability and lifetime. However, questions of where how many, and what types of heterogeneous resources to deploy remain largely unexplored. We focus on energy and link heterogeneity in ad hoc sensor networks and consider resource-aware MAC and routing protocols to utilize those resources. Using analysis, simulation, and real testbed measurements, we evaluate the impact of number and placement of heterogeneous resources on performance in networks of different sizes and densities. While we prove that optimal deployment is very hard in general, we also show that only a modest number of reliable, long-range backhaul links and line-powered nodes are required to have a significant impact. Properly deployed, heterogeneity can triple the average delivery rate and provide a 5-fold increase in the lifetime (respectively) of a large batten-powered network of simple sensors.
17852912
Lock Free Data Structures Using STM in Haskell
This paper explores the feasibility of re-expressing concurrent algorithms with explicit locks in terms of lock free code written using Haskell's implementation of software transactional memory. Experimental results are presented which show that for multi-processor systems the simpler lock free implementations offer superior performance when compared to their corresponding lock based implementations.
668272
First Results in the Coordination of Heterogeneous Robots for Large-Scale Assembly
While many multi-robot systems rely on fortuitous cooperation between agents, some tasks, such as the assembly of large structures, require tighter coordination. We present a general software architecture for coordinating heterogeneous robots that allows for both autonomy of the individual agents as well as explicit coordination. This paper presents recent results with three robots with very different configurations. Working as a team, these robots are able to perform a high-precision docking task that none could achieve individually.
32064388
Password sharing: implications for security design based on social practice
Current systems for banking authentication require that customers not reveal their access codes, even to members of the family. A study of banking and security in Australia shows that the practice of sharing passwords does not conform to this requirement. For married and de facto couples, password sharing is seen as a practical way of managing money and a demonstration of trust. Sharing Personal Identification Numbers (PINs) is a common practice among remote indigenous communities in Australia. In areas with poor banking access, this is the only way to access cash. People with certain disabilities have to share passwords with carers, and PIN numbers with retail clerks. In this paper we present the findings of a qualitative user study of banking and money management. We suggest design criteria for banking security systems, based on observed social and cultural practices of password and PIN number sharing.