18315245
An Environment Model for Nonstationary Reinforcement Learning
Reinforcement learning in nonstationary environments is generally regarded as an important and yet difficult problem. This paper partially addresses the problem by formalizing a subclass of nonstationary environments. The environment model, called hidden-mode Markov decision process (HM-MDP), assumes that environmental changes are always confined to a small number of hidden modes. A mode basically indexes a Markov decision process (MDP) and evolves with time according to a Markov chain. While HM-MDP is a special case of partially observable Markov decision processes (POMDP), modeling an HM-MDP environment via the more general POMDP model unnecessarily increases the problem complexity. A variant of the Baum-Welch algorithm is developed for model learning requiring less data and time.
3513935
A dynamic mechanism for time-constrained trading
The market mechanism design is important for both conventional and electronic commerce as it affects the fairness and efficiency of trading. In this paper, we propose a new market mechanism for time- constrained trading. Our mechanism mimics the traditional brokering system, where buyers and sellers meet together and negotiate through an electronic marketplace. At each time step, agents are paired up for negotiating a deal. We consider that buyer and seller agents are associated with a time constraint and a search cost, and thus must decide promptly and judiciously whether to accept an offer. One distinguishing feature of our mechanism is that the marketplace provides buyer and seller agents with statistical information about the goods. Specifically, the statistics include the probability distributions of obtaining and losing a particular offer. With this information, the agents' decision can be viewed as a Markov decision process, and the optimal (dominant) trading strategy can be computed. In other words, buyer and seller agents can make trading decisions that maximize their expected utility without the need of speculating others actions. We also explain how such a dominant strategy can be computed in an efficient manner. Another distinguishing feature of the proposed mechanism is that the statistics are updated continuously and hence the dominant trading strategy is adaptive. Experimental results verify that our mechanism facilitates a fair allocation to the trading agents under various demand and supply conditions.
858957
Hidden-Mode Markov Decision Processes for Nonstationary Sequential Decision Making
Problem formulation is often an important first step for solving a problem effectively. In sequential decision problems, Markov decision process (MDP) ([2]; [22]) is a model formulation that has been commonly used, due to its generality, flexibility, and applicability to a wide range of problems. Despite these advantages, there are three necessary conditions that must be satisfied before the MDP model can be applied; that is, 
 
1. 
 
The environment model is given in advance (a completely-known environment). 
 
 
 
 
2. 
 
The environment states are completely observable (fully-observable states, implying a Markovian environment). 
 
 
 
 
3. 
 
The environment parameters do not change over time (a stationary environment).
1688268
Mobile Agent Based Adaptive Scheduling Mechanism in Peer to Peer Grid Computing
In a peer to peer grid computing environment, volunteers are exposed to failures such as crash and link failures. In addition, since volunteers can dynamically join and leave executions and they are not dedicated only to a peer to peer grid computing, the executions of volunteers are stopped or suspended more frequently than in a grid computing environment. These failures result in the delay and blocking of the executions of tasks and even partial or entire loss of the executions. In addition, these failures make it difficult for a volunteer server to schedule tasks and manage the allocated tasks as well as volunteers. Existing peer to peer grid computing systems, however, do not deal with these failures in scheduling mechanisms. Moreover, since existing scheduling mechanisms are performed only by a volunteer server in a centralized way, there is a high overhead. 
 
To solve these problems, we propose a mobile agent based adaptive scheduling mechanism (MAASM). We implemented MAASM in Korea@Home and ODDUGI mobile agent system. The MAASM reduces the overhead of volunteer server by using mobile agents in scheduling procedure in a distributed way. In addition, it tolerates the various failures(especially, volunteer autonomy failures) which frequently occur in a peer to peer grid computing environment. Consequently, MAASM guarantees reliable and continuous executions in spite of the failures, so it decreases total execution time.
26521398
Advanced Stochastic Host State Modeling to Reliable Computation in Global Computing Environment
To design a stable global computing environment supporting reliable job execution and covering unanticipated state changes of hosts, the dynamic characteristics (i.e. volatilities) of hosts should be considered. Since a host is not dedicated to the system, voluntary hosts are free to leave and join autonomously in the middle of execution. As current systems do not relate volatility to a scheduling procedure, global computing systems suffer from performance degradation, reliability loss, job interruption, and execution time delays. For dependable computation, we propose Advanced Stochastic Host State Modeling (ASHSM), which is based on Markov model relating to execution availability quantifying duration and regularity of execution patterns of each host. Through the model, the system predicts desktop activities and allocates jobs according to the host features. Furthermore ASHSM alleviates unreliability due to unstable resource provision and job suspension during execution
29438564
Motion-compensated 3-D subband coding of video
This paper describes a video coding system based on motion-compensated three-dimensional (3-D) subband/wavelet coding (MC-3DSBC), which can overcome the limits of both 3-D SBC and MC prediction-based coding. In this new system, spatio-temporal subbands are generated by MC temporal analysis and a spatial wavelet transform, and then encoded by 3-D subband-finite state scalar quantization (3DSB-FSSQ). The rate allocation from the GOP level to each class of subbands is optimized by utilizing the structural property of MC-3DSBC that additive superposition approximately holds for both rate and distortion. The proposed video coding system is applied to several test video clips. Its performance exceeds that of both a known MPEG-1 implementation and a similar subband MC predictive coder while maintaining modest computational complexity and memory size.
5224987
Differential learning algorithms for decorrelation and independent component analysis
Decorrelation and its higher-order generalization, independent component analysis (ICA), are fundamental and important tasks in unsupervised learning, that were studied mainly in the domain of Hebbian learning. In this paper we present a variation of the natural gradient ICA, differential ICA, where the learning relies on the concurrent change of output variables. We interpret the differential learning as the maximum likelihood estimation of parameters with latent variables represented by the random walk model. In such a framework, we derive the differential ICA algorithm and, in addition, we also present the differential decorrelation algorithm that is treated as a special instance of the differential ICA. Algorithm derivation and local stability analysis are given with some numerical experimental results.
16841890
Scheduling Scheme based on Dedication Rate in Volunteer Computing Environment
A volunteer node can join and leave a volunteer computing system freely. However, existing volunteer computing systems suffer from interruptions of job execution, delays of execution time, and increases of total execution time, since they do not consider dynamic scheduling properties (i.e. volatilities), such as leave, join, and suspension. Therefore, dynamic execution properties of volunteer node should be considered in scheduling schemes, in order to design a stable and reliable volunteer computing system. This paper proposes a new scheduling scheme based on the dedication rate (DR), which reflects the dynamic properties of a volunteer. The scheduling scheme improves the completeness and reliability of execution, while also decreasing delay and total execution time. In addition, an implementation of the proposed scheduling scheme on top of Korea@Home is described, as well as performance evaluation