Infinite Horizon Optimal Transmission Power Control
for Remote State Estimation over Fading Channels
Xiaoqiang Ren, Junfeng Wu, Karl Henrik Johansson, Guodong Shi, and Ling Shi∗†‡

arXiv:1604.08680v1 [] 29 Apr 2016

Abstract
Jointly optimal transmission power control and remote estimation over an infinite horizon is studied. A sensor observes a dynamic process and sends its observations to a remote estimator over a
wireless fading channel characterized by a time-homogeneous Markov chain. The successful transmission probability depends on both the channel gains and the transmission power used by the sensor.
The transmission power control rule and the remote estimator should be jointly designed, aiming to
minimize an infinite-horizon cost consisting of the power usage and the remote estimation error. A first
question one may ask is: Does this joint optimization problem have a solution? We formulate the joint
optimization problem as an average cost belief-state Markov decision process and answer the question
by proving that there exists an optimal deterministic and stationary policy. We then show that when
the monitored dynamic process is scalar, the optimal remote estimates depend only on the most recently received sensor observation, and the optimal transmission power is symmetric and monotonically
increasing with respect to the innovation error.

1

Introduction

In networked control systems, control loops are often closed over a shared wireless communication network.
This motivates research on remote state estimation problems, where a sensor measures the state of a linear
system and transmits its observations to a remote estimator over a wireless fading channel. Such monitoring
problems appear in a wide range of applications in environmental monitoring, space exploration, smart
grids, intelligent buildings, among others. The challenges introduced by the networked setting lie in the
fact that nonideal communication environment and constrained power supplies at sensing nodes may result
in overall system performance degradation. The past decade has witnessed tremendous research efforts
devoted to communication-constrained estimation problems, with the purpose of establishing a balance
between estimation performance and communication cost.
∗

X. Ren and L. Shi are with the Department of Electronic and Computer Engineering, The Hong Kong University of

Science and Technology, Hong Kong. Email: xren@connect.ust.hk, eesling@ust.hk
†
J. Wu and K. H. Johansson are with the ACCESS Linnaeus Center, School of Electrical Engineering, Royal Institute of
Technology, Stockholm, Sweden. Email: junfengw@kth.se, kallej@kth.se
‡
G. Shi is with College of Engineering and Computer Science, The Australian National University, Canberra, Australia.
Email: guodong.shi@anu.edu.au

1

1.1

Related Work

Wireless communications are being widely used nowadays in sensor networks and networked control systems. The interface of control and wireless communication has been a central theme in the study of
networked sensing and control systems in the past decade. Early works assumed finite-capacity digital
channels and focused on the minimum channel capacity or data rate needed for feedback stabilization,
and on constructing encoder-decoder pairs to improve performance, e.g., [1–5]. Motivated by the fact that
packets are the fundamental information carrier in most modern data networks [6], networked control and
estimation subject to packet delays [7–9] and packet losses [10–13] has been extensively studied.
State estimation is embedded in many networked control applications, playing a fundamental role
therein. For networked state estimation subject to limited communication resource, the research on controlled communication has been extensive, see the survey [6]. Controlled communication, in general referring to reducing the communication rate intentionally to obtain a desirable tradeoff between the estimation
performance and the communication rate, is motivated from at least two facts: (i). Wireless sensors are
usually battery-powered and sparsely deployed, and replacement of battery is difficult or even impossible,
so the amount of communication needs to e kept at a minimum as communication is often the dominating
on-board energy consumer [14]. (ii). Traffic congestion in a sensor network many lead to packet losses
and other network performance degradation. To minimize the inevitable enlarged estimation error due to
reduced communication rate, a communication scheduling strategy for the sensor is needed. Two lines of
research directions are present in the literature. The first line is known as time-based (offline) scheduling, whereby the communication decisions are simply specified only according to the time. Informally, a
purely time-based strategy is likely to lead to a periodic communication schedule [15,16]. Optimal periodic
scheduling has been extensively studied, e.g, [17–20]. The second line is known as event-based scheduling,
whereby the communication decisions are specified according to the system state. The idea of event-based
scheduling was popularized by Lebesgue sampling [21]. Deterministic event-based transmission schedules
have been proposed in [22–31] for different application scenarios, and randomized event-based transmission schedules can be found in [32–34]. Essentially, event-based scheduling is a sequential decision problem
with a team of two agents (a sensor and an estimator). Due to the nonclassical information structure
of the two agents, joint optimization of the communication controller and the estimator is hard [35].
Most works [22–25, 27–31] bypassed the challenge by imposing restricted information structures or by
approximations, while some authors have obtained structural descriptions of the agents under the joint
optimization framework, using a majorization argument [26,28] or an iterative procedure [31]. In all these
works communication models were highly simplified, restricted to a binary switching model.
Fading is non-ignorable impairment to wireless communication [36]. The effects of fading has been taken
into account in networked control systems [37–40]. There are works that are concerned with transmission
power management for state estimation [41–47]. The power allocated to transmission affects the probability
of successful reception of the measurement, thus affecting the estimation performance. In [42], transmission
power is allocated via a predictive control algorithm based on the channel gain prediction. In [46], imperfect
2

acknowledgments of communication links and energy harvesting were taken into account. In [43], power
allocation for the estimation outage minimization problem was investigated in estimation of a scalar GaussMarkov source. In all of the aforementioned works, the estimation error covariances are a Markov chain
controlled by the transmission power, so Markov decision process (MDP) theory is ready for solving this
kind of problems. The reference [45] considered the case when plant state is transmitted from a sensor to
the controller over a wireless fading channel. The transmission power is adapted to the channel gain and
the plant states. Due to nonclassical information structure, joint optimization of plant input and transmit
power policies, although desired, is difficult. A restricted information structure was therefore imposed,
i.e., only a subset of the full information history available at the sensor is utilized when determining the
transmission power, to allow separate design at expense of loss of optimality. It seems that such a challenge
involved in these joint optimization problems always exists.

1.2

Contributions

In this paper, we consider a remote state estimation scheme, where a sensor measures the state of a
linear time-invariant discrete-time process and transmits its observations to a remote estimator over a
wireless fading channel characterized by a time-homogeneous Markov chain. The successful transmission
probability depends on both the channel gain and the transmission power used by the sensor. The objective
is to minimize an infinite horizon cost consisting of the power consumption and the remote estimation
error. In contrast to [45], no approximations are made to prevent loss of optimality, which however renders
the analysis challenging. We formulate our problem as an infinite horizon belief-state MDP with an average
cost criterion. Contrary to the finite horizon belief-state MDP considered in [28], for which an optimal
solution exists, a first question that one may ask about our infinite horizon MDP is: Does this optimization
problem have a solution? The answer is yes provided certain conditions given in this paper. On top of
this, we present structural results on the optimal transmission power controller and the remote estimator
for some special systems, which can be seen as the extension of the results in [26, 31] for the power
management scenario. The analysis tools used in the work (i.e., the partially observable Markov decision
process (POMDP) formulation and the majorization interpretation) is inspired by [28]. Nevertheless, the
contributions of the two works are distinct. In [28] the authors mainly studied the threshold structure
of the optimal communication strategy within a finite horizon, while the present work focuses on the
asymptotic analysis of the joint optimization problem over an infinite horizon.
In summary, the main contributions of this paper are listed as follows. We prove that a deterministic
and stationary policy is an optimal solution to the formulated average cost belief-state MDP. We should
remark that the abstractness of the considered state and action spaces (the state space is a probability
measure space and the action space a function space) renders the analysis rather challenging. Then we
prove that both the optimal estimator and the optimal power control have simple structures when the
dynamic process monitored is scalar. To be precise, the remote estimator synchronizes its estimates with
the data received in the presence of successful transmissions, and linearly projects its estimates a step
3

forward otherwise. For a certain belief, the optimal transmission power is a symmetric and monotonically
increasing function of the innovation error. Thanks to these properties, both the offline computation and
the online implementation of the optimal transmission power rule are greatly simplified, especially when
the available power levels are discrete, for which only thresholds of switchings between power levels are
to be determined.
This paper provides a theory in support of the study of infinite horizon communication-constrained
estimation problems. Deterministic and stationary policies are relatively easy to compute and implement,
thus it is important to know that an optimal solution that such a policy exists. The structural characteristic
of the jointly optimal transmission power and estimation policies provides insights into the design of
energy-efficient state estimation algorithms.

1.3

Paper Organization

In Section 2, we provide the mathematical formulation of the system model adopted, including the monitored dynamic process, the wireless fading channel, the transmission power controller and the remote
estimator. We then present the considered problem and formulate it as an average cost MDP in Section 3. In Section 4, we prove that there exists a deterministic and stationary policy that is optimal to the
formulated MDP. Some structural results about the optimal remote estimator and the optimal transmission power control strategy are presented in Section 5. Concluding remarks are given in Section 6. Some
auxiliary background results and a supporting lemma are provided in the appendices.

Notation
N and R+ are the sets of nonnegative integers and nonnegative real numbers, respectively. Sn+ (and Sn++ )
is the set of n by n positive semi-definite matrices (and positive definite matrices). When X ∈ Sn+ (and
Sn++ ), we write X  0 (and X ≻ 0). X  Y if X − Y ∈ Sn+ . Tr(·) and det(·) are the trace and the
determinant of a matrix, respectively. λmax (·) represents the eigenvalue, having the largest magnitude, of
a matrix. The superscripts

⊤

and

−1

stand for matrix transposition and matrix inversion, respectively.

The indictor function of a set A is defined as

 1, ω ∈ A
1A (ω) =
 0, ω ∈
6 A.

The notation p(x; x) represents the probability density function (pdf) of a random variable x taking value
at x. If being clear in the context, x is omitted. For a random variable x and a pdf θ, the notation x ∼ θ
means that x follows the distribution defined by θ. The symbol Nx0 ,Σ (x) denotes a Gaussian distribution
of x with mean x0 and covariance Σ. For measurable functions f, g : Rn 7→ R, we use f ∗ g to denote the
convolution of f and g. For a Lebesgue measurable set A ⊂ Rn , L(A) denotes the Lebesgue measure of A.
Let kxk denote the L2 norm of a vector x ∈ Rn . δij is the Dirac delta function, i.e., δij equals to 1 when
i = j and 0 otherwise.

4

Process

❄

xk✲ Wireless γk xk ✲

✲ Tx
✻
uk

Tx Power
Controller

Channel

Remote
Estimator

x̃k ✲

✛

Sensor
Figure 1: The remote state estimation scheme.

2

System Model

In this paper, we focus on dynamic power control for remote state estimation. We consider a remote
state estimation scheme as depicted in Figure 1. In this scheme, a sensor measures a linear time-invariant
discrete-time process and sends its measurement in the form of data packets, to a remote estimator over a
wireless link. The remote estimator produces an estimate of the process state based on the received data.
When sending packets through the wireless channel, transmissions may fail due to interference and weak
channel gains. Packet losses lead to distortion of the remote estimation and packet loss probabilities depend
on transmission power levels used by the transmitter and on the channel gains. Lower loss probabilities
require higher transmission power usage; on the other hand, energy saving is critical to expand the lifetime
of the sensor. The wireless communication overhead dominates the total power consumption, therefore
we introduce a transmission power controller, which aims to balance the transmission energy cost and
distortion penalty as the channel gain varies over time.
In what follows, the attention is devoted to laying out the main components in Figure 1.

2.1

State Process

We consider the following linear time-invariant discrete-time process:
xk+1 = Axk + wk ,

(1)

where k ∈ N, xk ∈ Rn is the process state vector at time k, wk ∈ Rn is zero-mean independent and identically distributed (i.i.d.) noises, described by the probability density function (pdf) µw , with E[wk wk′ ] = W
(W ≻ 0). The initial state x0 , independent of wk , k ∈ N, is described by pdf µx0 , with mean E[x0 ] and
covariance Σ0 . Without loss of generality, we assume E[x0 ] = 0, as nonzero-mean cases can be translated
into zero-mean one by coordinate change x′k = xk − E[x0 ]. We let X = Rn denote the domain of xk . The
system parameters are all known to the sensor as well as the remote estimator. We assume the plant is
unstable, i.e., |λmax (A)| > 1.

5

N0
2

gk
xk

N
✲ ❄

✲ Encoder

L
✲ ❄ ✲ Decoder

γk xk
✲

Wireless Channel

Figure 2: Wireless communication model. N0 is the power spectral density of the channel noise vk .

2.2

Wireless Communication Model

The sensor measures and sends the process state xk to the remote estimator over an additive white
Gaussian noise (AWGN) channel which suffers from channel fading (see Figure 2):
y = gk x + vk ,
where gk is a random complex number, and vk is additive white Gaussian noise; x represents the signal
(e.g., xk ) sent by the transmitter and y the signal received by the receiver. Let the channel gain hk = |gk |2
take values in a finite set h ⊆ R+ and {hk }k∈N possess temporal correlation modeled by a time-homogenous
Markov chain [40, 47]. The one-step transition probability for this chain is denoted by
π(·|·) : h × h 7−→ [0, 1].
The function π(·|·) is known a priori. We assume the remote estimator or the sensor can access the channel
state information (CSI), so the channel gain hk is available at each time before transmission. This can for
instance done using pilot-aided channel estimation techniques are adopted, by which the transmitter sends
a pilot signal at each fading block and the channel coefficients, including the channel gain, are obtained
at the receiver [36]. The estimation errors of the channel gains are not taken into account in this paper.

To facilitate our analysis, the following assumption is made.
Assumption 1 (Communication model).
(i). The channel gain hk is independent of the system paremeters.
(ii). The channel is block fading, i.e., the channel gain remains constant during each packet transmission
and varies from block to block.
(iii). The quantization effect is negligible and does not effect the remote estimator.
(iv). The receiver can detect symbol errors1 . Only the data reconstructed error-free are regarded as successfully reception. The receiver perfectly realizes whether the instantaneous communication succeeds
or not.
1

In practice, symbol errors can be detected via a cyclic redundancy check (CRC) code.

6

(v). The Markov chain governing the channel gains, π(·|·), is aperiodic and irreducible.
Assumption 1-(i)(ii)(iii)(iv) are standard for fading channel model. Note that Assumption 1-(i)(iii)(iv)
were used in [10, 44, 45, 48, 49], and that Assumption 1-(ii) was used in [42]. From Assumption 1-(iv),
whether or not the data sent by the sensor is successfully received by the remote estimator is indicated
by a sequence {γk }k∈N of random variables, where


1, if xk is received error-free at time k,
γk =

0, otherwise (regarded as dropout),

(2)

initialized with γ0 = 1. When γk = 0, we regard the remote estimator as having received a virtual symbol
♯. Assumption 1-(v) is a technical requirement for Theorem 1. One notes that both the i.i.d. channel gains
model and the Gilbert–Elliott model with the good/bad state transition probability not equal to 1 satisfies
Assumption 1-(v).

2.3

Transmission Power Controller

Let uk ∈ R+ be the transmission power at time k, the power supplied to the radio transmitter. Due to
constraints with respect to radio power amplifiers, the admissible transmission power is restricted. Let
uk take values in U ⊂ [0, umax ], in which umax stands for the maximum power. Depending on the radio
implementation, U may be a continuum or a finite set. It is further assumed that U is compact and
contains zero. Under Assumption 1-(iii), the successful packet reception is statistically determined by the
signal-to-noise ratio (SNR) hk pk /N0 at the receiver, where N0 is the power spectral density of vk . A very
general model [40, 45] of the conditional packet reception probabilities for a variety of modulations is as
follows:
q(uk , hk ) , P (γk = 1|uk , hk ) ,

(3)

where q is a nondecreasing function in both uk and hk .
Assumption 2. The function q(u, h) : U × h 7→ [0, 1] is continuous almost everywhere with respect to u
for any fixed h. Especially, q(0, h) = 0 for all h.
Remark 1. If letting q(uk , hk ) = q(uk ) with U = {0, 1} and

 1, if u = 1;
k
q(uk ) =
 0, if uk = 0,

we conclude that the “on-off ” controlled communication problem considered in [17–20, 22–34] is a special
case of the transmission power control problem considered here.
We assume that packet reception probabilities are conditionally independent for given channel gains
and transmission power levels, which is stated in the following assumption.

7

Assumption 3. The following equality holds for any k ∈ N,
P (γk = rk , . . . , γ1 = r1 |u1:k , h1:k ) =

k
Y

P (γj = rj |uj , hj ) .

j=1

Remark 2. Assumption 2 is standard for digital communication over fading channels. Assumption 3 is
in accordance with the common sense that the symbol error rate statistically depends on the instantaneous
SNR at the receiver. Many digital communication modulation methods are embraced by these assumptions [40, 42, 50, 51].
Assumption 4. For the least favorable channel power gain h , min{h : h ∈ h}, the maximum achievable
successful transmission probability satisfies
q(ū, h) > 1 −

1
,
λmax (A)2

where ū is the highest available power level: ū , max{u : u ∈ U } and A is the system matrix in (1).
Note that since both

h and U

are compact, h and ū always exist.

Remark 3. Assumption 4 provides a sufficient condition under which the expected estimation error covariance is bounded, even for the least favorable channel power gain. Similar assumptions were also adopted
in many works, such as [40, 46], for guaranteeing the stability of the Kalman filtering subject to random
packet losses.

2.4

Remote Estimator

At the base station side, each time a remote estimator generates an estimate based on what it has
received from the sensor. In many applications, the remote estimator is powered by an external source or
is connected with an energy-abundant controller/actuator, thus having sufficient communication energy in
contrast to the energy-constrained sensor. This energy asymmetry allows us to assume that the estimator
can send messages back to the sensor. The content of feedback messages are separatively defined under
different system implementations, the details of which will be discussed later in Section 3.2. Denote by
Ok− the observation obtained by the remote estimator up to before the communication at time k, i.e.,
Ok− , {γ1 x1 , ..., γk−1 xk−1 } ∪ {γ1 , . . . , γk−1 } ∪ {h1 , . . . , hk }.
Similarly, denote by Ok+ the observation obtained by the remote estimator up to after the communication
at time k, where
Ok+ , Ok− ∪ {γk , γk xk }.

3

Problem Definition

We take into account both the estimation quality at the remote estimator and the transmission energy
consumed by the sensor. To this purpose, joint design of the transmission power controller and the remote
8

estimator is desired. Measurement realizations, communication indicators, and channel gains are adopted
to manage the usage of transmission power:

uk = fk x1:k , h1:k , γ1:k−1 .

(4)

To produce uk , the remote estimator can work under two different implementations, regarding the computational capacity of the sensor node: the remote estimator decides an intermediate function
lk (·) = fk (Ok− , ·)

(5)

and feeds lk back to the sensor and then the sensor evaluates uk by uk = lk (x1:k ); or, equivalently, the
estimator directly feeds γk ’s back to the sensor, and the sensor is in charge of deciding fk . Given the
transmission power controller, the remote estimator generates an estimate as a function of what it has
received from the sensor, i.e.,
x̃k , gk (Ok+ ).

(6)

We emphasize that x̃k also depends on fk since fk statistically affects the arrival of the data. The average
remote estimation quality over an infinite time horizon is quantified by


T
X
1
kxk − x̃k k2  ;
E(f , g) , Ef ,g limsup
T
T →∞

(7)

j=1

correspondingly, the average transmission power cost, denoted as W (f , g), is given by
#
"
T
1X
uk ,
W (f ) , Ef limsup
T →∞ T

(8)

k=1

where f , {f1 , . . . , fk , . . .} and g , {g1 , . . . , gt , . . .}. The arguments f and g indicate that the quantity
of (7) depends on them. This is also the case in (8). Note that in (7) and (8) the expectations are taken
with respect to the randomness of the system and the transmission outcomes for given f and g. For the
remote state estimation system, we naturally wonder how to find a jointly optimal transmission power
controller fk∗ and remote state estimator gk∗ satisfying
minimizef ,g [E(f , g) + α W (f )] ,

(9)

where the constant α can be interpreted as a Lagrange multiplier. We should remark that (9) is difficult
to solve due to the nonclassical information structure [35]. What is more, (9) has an average cost criterion
that depends only on the limiting behavior of f and g, adding additional analysis difficulty.

3.1

Belief-State Markov Decision Process

To find a solution to the optimization problem (9), we first observe from (8) that W (f ) does not depend on
g, thus leading to an insight into the structure of gk∗ —Lemma 1, the proof of which follows from optimal
filtering theory: the conditional mean is the minimum-variance estimate [52]. Similar results can be seen
in [26, 28, 31].
9

Lemma 1. For any given transmission power controller fk , the optimal remote estimator gk∗ is the MMSE
estimator
x̂k , gk∗ (Ok+ ) = Ef1:k [xk |Ok+ ].

(10)

Problem (9) still remains hard since gk∗ depends on the choice of f1:k . To address this issue, by viewing the
problem from the perspective of a decision maker holding the common information [53], we formulate (9)
as a POMDP [54] at the decision maker’s side. Following the conventional treatment of the POMDP, we
are allowed to equivalently study its belief-state MDP [54]. For technical reasons, we pose two moderate
constraints on the action space. We will present the formal belief-state MDP model and remark that the
resulting gap between the formulated belief-state MDP and (9) is negligible (see Remark 6). Before doing
so, a few definitions and notations are needed. Define innovation ek as
ek , xk − Ak−τ (k) xτ (k)

(11)

with ek taking values in Rn and τk being the most recent time the remote estimator received data before
time k as
τ (k) , max {t : γt = 1}.
16t6k−1

(12)

+
, the equality
Let êk , Ef1:k [ek |Ok+ ]. Since τ (k), xτ (k) ∈ Ok−1

ek − êk = xk − x̂k

(13)

+
.
holds for all k ∈ N. In other words, ek can be treated as xk offset by a variable that is measurable to Ok−1

We define the belief state on ek . From (13), the belief state on xk can be equally defined. Here we use ek
instead of xk for ease of presentation.
Definition 1. Before the transmission at time k, the belief state θk (·) : Rn 7→ R+ is defined as θk (e) ,
+
).
p(ek ; e|f1:k , Ok−1

We also need some definitions related to a partition of a set.
Definition 2. A collection ∆ of sets is a partition of a set X if the following conditions are satisfied:
(i). ∅ 6∈ ∆.
(ii). ∪B∈∆ B = X .
(iii). If B1 , B2 ∈ ∆ and B1 6= B2 , then B1 ∩ B2 = ∅.
An element of ∆ is also called a cell of ∆. If X ⊂ Rn , we define the size of ∆ as
|∆| , sup {kx − yk : x, y ∈ δj , δj ∈ ∆}.
δj ,x,y

Definition 3. For two partitions, denoted as ∆1 and ∆2 , of a set X , ∆1 is called a refinement of ∆2 if
every cell of ∆1 is a subset of some cell of ∆2 . Formally it is written as ∆1  ∆2 .
10

One can verify that the relation  is a partial order, and the set of partitions together with this relation
form a lattice. We denote the infimum of partitions ∆1 and ∆2 as ∆1 ∧ ∆2 .
Now we are able to mathematically describe the belief-state MDP by a quintuplet (N, S, A, P , C ). Each
item in the tuple is elaborated as follows. We provide background knowledge of the mathematical notions
used below in the Appendix A.
(i). The set of decision epochs is N.
(ii). State space S = Θ × h: Θ is the set of beliefs over Rn , i.e., the space of probability measures on Rn .
The set Θ is further constrained as follows. Let µ be a generic element of Θ. Then µ is equivalent to
R
the Lebesgue measure2 , and µ has the finite second moment, i.e., Rn kek2 dµ(e) < ∞. No generality

has been lost by the above two constraints; see Remark 4. Let θ(e) =

dµ(e)
dL(e)

be the Radon–Nikodym

derivative [55]. Note that θ(e) is uniquely defined up to a L−null set (i.e., a set having Lebesgue
measure zero). We thus use µ and θ(e) interchangeably to represent a probability measure on Rn ,
and we do not distinguish between any two functions θ(e) and θ ′ (e) with L({e : θ(e)−θ(e)′ 6= 0}) = 0
by convention. We assume that Θ is endowed with the topology of weak convergence. Denote by

dP (·, ·) denote the Prohorov metric [56] on Θ. We define the
as ds ((µ1 , h1 ), (µ2 , h2 )) = max{dP (µ1 , µ2 ), |h1 − h2 |}.

s , (µ, h) a generic element of S. Let
metric on S

(iii). Action space A is the set of all functions that have the following structure:

 ū,
if kek > L,
a(e) =
 a′ (e), otherwise,

(14)

where a′ ∈ A′ : E 7→ U with E , {e ∈ Rn : kek ≤ L}. The space A′ is further defined as follows.
Let a′ ∈ A′ be a generic element, then there exists a finite partition ∆a′ of E such that each cell

of ∆a′ is a L−continuity set3 and on each cell a′ (e) is Lipschitz continuous with Lipschitz constant
uniformly bounded by M . It is further assumed that ∆ = ∧a′ ∈A′ ∆a′ is a finite partition of E. Since
both L and M can be arbitrarily large and |∆| can be arbitrarily small, the structure constraints
pose little limitation; see Remark 5 why we consider such an action space. We consider the Skorohod
distance defined in (38). By convention, we do not distinguish two functions in A that have zero
distance and we consider the space of the resulting equivalence classes. Note that the argument of
the function a(·) is the innovation ek defined in (11), and by the definition of ek , one obtains that
ak (e) = lk (e + Ak−τ (k) xτ (k) ).
(iv). The function P (θ ′ , h′ |θ, h, a) : S × A × S defines the conditional state transition probability. To be
2

Let µ1 and µ2 be measures on the same measurable space. Then µ1 is said to be equivalent to µ2 if for any Borel subset B,

µ2 (B) = 0 ⇔ µ1 (B) = 0.
3
A Borel subset B is said to be a µ−continuity set if µ(∂B) = 0, where ∂B is the boundary set of B.

11

precise,
P (θ ′ , h′ |θ, h, a)
, p(θk+1 , hk+1 ; θ ′ , h′ |θk = θ, hk = h, ak = a)

′
′
′


 π(h |h) (1 − ϕ(θ, h, a)) δφ(θ,h,a,0) (θ ), if θ = φ(θ, h, a, 0),
π(h′ |h)ϕ(θ, h, a)δφ(θ,h,a,1) (θ ′ ), if θ ′ = φ(θ, h, a, 1),

=





0,

otherwise,

R
where ϕ(θ, h, a) , Rn q(a(e), h)θ(e)de, δθ∗ (θ) denotes a degenerate distribution at θ∗ over Θ with
R
Θ δθ∗ (θ)dθ = 1, and

+
where θθ,h,a
(e) ,

φ(θ, h, a, γ)

+
1
−1

det(A) θθ,h,a (A e) ∗ N0,W (e), if γ = 0,
,
 N0,W (e),
if γ = 1,

(1−q(a(e),h))θ(e)
1−ϕ(θ,h,a)

(15)

is interpreted as the post-transmission belief when the transmission

fails, and N0,W (e) is the multivariate Gaussian distribution with mean 0 and covariance W .
(v). The function C (θ, h, a) : S × A → R+ is the cost function when performing a ∈ A for θ ∈ Θ and
h ∈ h at time k, which is given by
C (θ, h, a) =

Z

θ(e)c(e, h, a)de.

(16)

Rn

In (16), the function c(·, ·, ·) : Rn × h × A 7→ R+ is defined as c(e, h, a) = αa(e) + (1 − q(a(e), h))ke −
ê+ k2 with ê+ = Eθ+

θ,h,a

+
], where the communication cost is counted by the first
[e] , E[e|e ∼ θθ,h,a

term and the distortion ke − ê+ k2 with probability 1 − q(a(e), h) is counted by the second term.
Remark 4. The initial belief θ1 (e) = 1/det(A)µx0 (A−1 e) ∗ N0,W (e) is equivalent to the Lebesgue measure.
The belief evolution in (15) gives that, whatever policy is used, θk is equivalent to the Lebesgue measure
for k ≥ 2. Also, note that if there exists a channel gain h ∈

h such that q(ū, h) < 1 and if θ

has infinite

second moment, then C (θ, h, a) = ∞ for any action a. Thus, to solve (9), without any performance loss,
we can restrict beliefs into the state space Θ.
Remark 5. The action a(e) ∈ A is allowed to have a L−null set of discontinuity points. The assumption
that on each cell of a partition, a(e) is a Lipschitz function is a technical requirement for Theorem 1. The
intuition is that given θk , except for L−null set of points, the difference between the power used for ek
and e′k is at most proportional to the distance between ek and e′k . The saturation structure in (14), i.e.,
a(e) = ū if kek > L is also a technical requirement for Theorem 1. Intuitively, this ensures that, when the
+
transmission fails, the second moment of the post-transmission belief θθ,h,a
(e) is bounded by a function of

the second moment of θ(e). The saturation assumption can also be found in [45].

12

✻
x̃k

γk x k ✲
θk ✲
Estimator
Observation
acqui. & trans.

MDP
algorithm

Remote Estimator
✛

action lk
(a)
Process
xk
❄
Tx Power
Controller

u✲
k

Sensor
Tx

xk✲

Transmission
& Estimation

x̃k ✲

✻
l

k

MDP
algorithm

✛θk

γk

Virtual ✛
estimation

✻

(b)
Figure 3: Implementation of the system. The block “Observation acqui. & trans.” in (a) corresponds to the
blue-dashed rectangle in Figure 1 and the block “Transmission & estimation” the red-dashed rectangle. In
(a), the MDP algorithm is run at the remote estimator and the action lk is fed back to the sensor. While
in (b), the MDP algorithm is run at the sensor node and γk is fed back by the remote estimator.

An admissible k−history for this MDP is defined as hk , {θ1 , h1 , a1 , . . . , θk−1 , hk−1 , ak−1 , θk , hk }. Let
Hk denote the class of all the admissible k−history hk . A generic policy d for (N, S, A, P , C ) is a sequence
of decision rules {dk }k∈N , with each dk : Hk → A. In general, dk may be a stochastic mapping. Let D
denote the whole class of d. In some cases, we may write d as d(dk ) to explicitly point out the decision
rules used at each stage. We focus on the following problem.
Problem 1. Find optimal policy for the MDP problem:
minimize J (d, θ, h),
d∈D

∀ (θ, h) ∈ S,

(17)

where
T
1 X θ,h
Ed [C (θk , hk , ak )]
J (d, θ, h) , limsup
T →∞ T
k=1

is the average cost with the initial state (θ, h) and the policy d.
Remark 6. The gap between Problem 1 and the optimization problem (9) arises from the structure
assumptions for the action space. These structure constraints, however, are moderate, since the saturation
level L and the uniform Lipschitz constant M can be arbitrarily large and the size of |∆| can be arbitrarily
small.

13

3.2

Practical Implementation

Here we discuss about the implementation of the system, which is illustrated in Figure 3. Depending
on the computational capacity of the senor node, the system we study can work either as in (a) or in
(b). The main difference between the systems in (a) and (b) is where the MDP algorithm is run. The
computational capacity required for the sensor node as well as the content of feedback messages are
correspondingly different. In (a), the MDP algorithm is run at the remote estimator and the action lk
is fed back to the sensor. In practice, for a generic lk , only an approximate version (e.g., lookup tables)
can be transmitted due to bandwidth limitation. An accurate feedback of lk is possible if lk has a special
structure. For example, if ak (e) (recall that that ak (e) = lk (e + Ak−τ (k) xτ (k) )) is a monotonic step function
taking values in a finite set, only those points, where ak jumps, are needed to represent lk (note that
Ak−τ (k) xτ (k) is available at the sensor node). Since the function lk is directly fed back to the sensor, the
only computational task carried out by the sensor is computing lk (xk ). When the sensor node is capable of
running the MDP algorithm locally, the system can be implemented as illustrated in (b). In this case, only
γk (a binary variable) is fed back. Note that when γk is fed back, the sensor knows exactly the information
available at the remote estimator. It can run a virtual estimator locally that has the same behavior as the
remote estimator.

4

Optimal Deterministic Stationary Policy: Existence

The definition of the policy d in the above section allows the dependence of dk on the full k−history,
hk . Fortunately, with the aid of the results of average cost MDPs [57–59], we prove that there exists
a deterministic stationary policy that is optimal to Problem 1. Before showing the main theorem, we
introduce some notations.
We define the class of deterministic and stationary policies Dds as follows: d(dk ) ∈ Dds if and only if
there exists a Borel measurable function d : S 7→ A such that ∀i,
dk (Hk−1 , ak−1 , θk = θ, hk = h) = d(θ, h).
Since the decision rules dk ’s are identical (equal d) along the time horizon for a stationary policy d({dk }k∈N ) ∈
Dds , we write it as d(d) for the ease of notation.
Theorem 1. There exists a deterministic and stationary policy d∗ (d) ∈ Dds such that
J (d∗ (d), θ, h) ≤ J (d, θ, h)

∀ (θ, h) ∈ S, d ∈ D,

Moreover,
d∗ (d) = arg min{Cd (θ, h) − ρ∗ (θ, h) + Ed [Q ∗ (θ ′ , h′ )|θ, h},
d∈Dds

and
J (d∗ (d), θ, h) = ρ∗ (θ, h),
14

(18)

where the functions Q ∗ : S 7→ R and ρ∗ : 7→R satisfy
Q ∗ (θ, h) = min {Cd (θ, h) − ρ∗ (θ, h) + Ed [Q ∗ (θ ′ , h′ )|θ, h]}
d∈Dds

with Cd (θ, h) , C (θ, h, d(θ, h)) and Ed [Q ∗ (θ ′ , h′ )|θ, h] ,

R

S

Q ∗ (θ ′ , h′ )P (θ ′ , h′ |θ, h, d(θ, h))d(θ ′ , h′ ).

The above theorem says that the optimal power transmission policy exists and is deterministic and
stationary, i.e., the power used at the sensor node uk only depends on (θk , hk ) and ek . Since the belief
state θk can be updated recursively as in (15), this property facilitates the related performance analysis.
In principle, the optimal deterministic and stationary policy to an average cost MDP can be obtained
by the value iteration algorithm [59]. However, it is not computationally tractable to solve (18), since
neither the state space nor the action space is finite. An approximate algorithm is needed to obtain an
suboptimal solution [54], which is out of the scope of this paper. Nevertheless, Theorem 1 provides a
qualitative characteristic of the optimal transmission power control rule.
Define
Jβ (d, θ, h) ,

Edθ,h

"∞
X

k

#

β C (θk , hk , ak )

k=1

as the expected total discounted cost with the discount factor 0 < β < 1. Let υβ (θ, h) , inf d∈D Jβ (d, θ, h)
be the least cost associated with the initial state (θ, h), and let mβ = inf (θ,h)∈S υβ (θ, h).
By Theorem 3.8 in [57], in order to prove Theorem 1, it is sufficient to verify the following conditions.
C1 (State Space) The state space S is locally compact with countable base.
C2 (Regularity) Let M be a mapping assigning to each s ∈ S the nonempty available action space A(s).
Then for each s ∈ S, A(s) is compact, and M is upper semicontinuous.
C3 (Transition Kernel) The state transition kernel P (·|s, a) is weakly continuous4 .
C4 (Cost Function) The one stage cost function C (s, a) is lower semicontinuous.
C5 (Relative Discounted Value Function) There holds
sup [υβ (θ, h) − mβ ] < ∞, ∀(θ, h) ∈ S.

(19)

0<β<1

We now verify each of the above conditions for the considered problem, by which we establish the proof
of Theorem 1.
4

We say P (·|s, a) is is weakly continuous if as i → ∞,
Z
Z
b(s′ )P (ds′ |si , ai ) →
b(s′ )P (ds′ |s, a)
S

S

for any sequence {(si , ai ), i ≥ 1} converging to (s, a) with si , s ∈ S and ai , a ∈ A, and for any bounded and continuous
function b : S 7→ R.

15

4.1

State Space Condition C1

We prove that both S and A are Borel subsets of Polish spaces (i.e., separable completely metrizable
topological spaces) instead. Then as pointed out in [58], by Arsenin–Kunugui Thoerem, the condition C1
holds.
To show that S is a Borel subset of a Polish space, by the well known results about the product
topology [60], it suffices to prove that Θ and

h are Borel subsets of Polish spaces. Since h is a compact

subset of R, we only need to prove Θ is a Borel subset of a Polish space. Let M(Rn ) be the space of
probability measures on Rn endowed with the topology of weak convergence. It is well known that M(Rn )
is a Polish space [56]. Let M2 (Rn ) ⊆ M(Rn ) be the set of probability measures with finite second moment,
and Me (Rn ) ⊆ M(Rn ) be the set of probability measures equivalent to L. By Theorem 3.5 in [61], Me (Rn )
w

is a Borel set. We then show that M2 (Rn ) is closed. Suppose {µi,i∈N } ∈ M2 (Rn ) and µi → µ. Since M(Rn )
is complete, µ ∈ M(Rn ), and using the fact that norms are continuous, by Theorem 1.1 in [62],
Z
Z
2
kek2 µi (de) < ∞.
kek µ(de) ≤ lim inf
i→∞

Rn

Then µ ∈ M2

(Rn ),

implying that M2

(Rn )

Rn

is closed. Since Θ = M2 (Rn ) ∩ Me (Rn ), Θ is a Borel subset

of M(Rn ). The state space S thus is a Borel subset of a Polish space.
Now we shall show that A is a Borel subset of a Polish space. Since a bounded function can be
approximated by simple functions uniformly [63], the space A′ is a subset of the general Skorohod space
defined on E (see Appendix A), i.e., A′ ⊆ D(E). We first prove that the closure of A′ , denoted as cl(A′ ),
is a compact set by Theorem 3.11 in [64]. Since a generic a ∈ A′ maps from E to [0, ū], the condition 3.37
is obviously satisfied. Note that the condition 3.38 is equivalent to
lim sup

∆ a∈A′

w(a, ∆) → 0.

(20)

By the definition of ∆, all the functions in A′ are Lipschitz continuous with Lipschitz constant uniformly
bounded by M on each cell of ∆. Thus, for ∆  ∆,
sup
a∈A′

w(a, ∆) ≤ M |∆|,

which yields (20). We then show that cl(A′ ) = A′ . Suppose that ai ∈ A′ converges to a limit a in the
s

Skorohod topology (we write as ai → a), we then show that a ∈ A′ . By the definition of the Skorohod
distance d(·, ·) in (38), ai → a if and only if there exist mappings πi ∈ Λt such that
s

lim ai (πi x) = a(x) uniformly in E
i

(21)

and limi πi x = x uniformly in E. Since limi πi x = x uniformly in E, for any ǫ > 0, there exists i0 such that
kπi kt < ǫ with i ≥ i0 . Note that if kπi kt < ǫ, πi is a bi-Lipschitz homeomorphism. By the definition of A′ ,
any ai ∈ A′ has L−null set of discontinuity points. Since measure-null sets are preserved by a Lipschitz
homeomorphism, by (21), one obtains that
L(the set of discontinuity points of a) = 0.
16

(22)

Following the same reasoning for one dimensional Skorohod space D[0, 1] (see e.g., P124 in [56]), one
s

obtains that ai → a implies that ai (x) → a(x) uniformly for all continuity points x of a. Since on each
cell of ∆ = {δj }, all the functions in A′ are Lipschitz continuous, the interior points of δj (write the set
as δjo ) must be continuity points of a. By the fact that if a sequence of Lipschitz functions with Lipschitz
constant uniformly bounded by M converge to a limit function, then this limit function is also a Lipschitz
function with Lipschitz constant bounded by the same M , a is Lipschitz continuous with Lipschitz constant
uniformly bounded by M on the interior set of each cell of ∆. For a boundary point x of the cells of ∆,
denote the collection of cells whose boundary contains x as δx , {δj : x ∈ ∂δj }. Then one obtains that a(x)
must be a limit of a from one cell in δx , i.e., there exists δj ∈ δx such that limy→x,y∈δjo a(y) = a(x). Now we
define a function a∗ such that for each δj ∈ ∆, a∗ (x) = a(x) if x ∈ δjo and a∗ (x) are continuous on δj . Then
one obtains that

d(a, a∗ ) = 0, which implies that a = a∗

since D(E) is a metric space. Combining (22),

one obtains that a ∈ A′ . Thus A′ is closed and compact. Using the fact that every compact metric space
is complete and separable, one obtains that A′ is a Polish space. By the structure relation between A and
A′ in (14), the space A is also a Polish space.

4.2

Regularity Condition C2

Since A is compact and A(s) = A for every s ∈ S, C2 is readily verified.

4.3

Transition Kernel Condition C3

Since S is separable and given (θk , hk , a), hk+1 and θk+1 are independent, then by Theorem 2.8 in [56], it
suffices to prove that for any h ∈ h, as θi → θ (µi → µ) and ai → a, the followings hold:
w

w

s

ϕ(θi , h, ai ) → ϕ(θ, h, a)

(23)

w

and φ(θi , h, ai , 0) → φ(θ, h, a, 0).

(24)

s

Since ai → a implies that ai (x) → a(x) uniformly for all the continuity points of a and the set of
s

discontinuity points of a has Lebesgue measure zero, ai → a implies that ai → a L−a.e. Noting that µ is
equivalent to L, ai → a µ−a.e. holds, and it follows that q(ai , h) → q(a, h) µ−a.e., since q is continuous
sw

L−a.e. Also, by Lemma 4, µi → µ. Then by Theorem 2.2 in [65], one obtains that
Z
Z
q(a(e), h)µ(de)
q(ai (e), h)µi (de) ≥
lim inf
i→∞
n
n
R
R
Z
Z
q(a(e), h)µ(de).
−q(ai (e), h)µi (de) ≥ −
and lim inf
i→∞

Rn

Rn

Combing the above two equations, one obtains that limi→∞
i.e., ϕ(θi , h, ai ) → ϕ(θ, h, a).

R

Rn

q(ai (e), h)µi (de) →

R

Rn

q(a(e), h)µ(de),

sw

We now prove that the equation (24) holds. Noting that θi → θ implies that θi (e)→θ(e) L−a.e., it thus
follows that
+
(e)
θθ+i ,h,ai (e) → θθ,h,a

17

(25)

+
L−a.e. Note that θθ+i ,hi ,ai (e) and θθ,h,a
(e) can be viewed as probability density functions of e, and for
+
simplicity, we write the corresponding probability measures as µ+
i and µ , respectively. Then it follows

from (25) that
sw

+
µ+
i →µ .

(26)

Let b(e) be any bounded and continuous function defined on Rn , then
Z
b(e)φ(θ, h, a, 0)(e)de
RnZ
Z
+
θθ,h,a
(e′ )N0,W (e − Ae′ )de′ de
b(e)
=
n
n
R Z
ZR
+
b(e)N0,W (e − Ae′ )dede′
θθ,h,a(e′ )
=
n
n
R
ZR
′ +
b̃(e )µ (de′ ),
,
Rn

where b̃(e′ ) ,

R

Rn

b(e)N0,W (e − Ae′ )de. Noting that b̃(e′ ) is a bounded function, then by Appendix E

of [59] and (26),

Z

′

b̃(e
Rn

′
)µ+
i (de )

→

Z

b̃(e′ )µ+ (de′ ).
Rn

The equation (24) thus follows by the Portmanteau Theorem.

4.4

Cost Function Condition C4

We first show that µ+ also has finite second moment given that µ has finite second moment.
Z
kek2 dµ+ (e)
n
R
Z
Z
kek2 dµ+ (e)
kek2 dµ+ (e) +
=
n
R \E
E
Z
kek2 dµ(e), for any h, a
≤ L2 +
Rn \E

< ∞,
+
where the first inequality follows from the structure of a(e) in (14). Since θθ,h,a
has finite second moment,
+
is uniformly integrable. Then by Theorem 3.5 in [56],
e ∼ θθ,h,a

êi+ → ê+
where êi+ = Eθ+

θi ,h,ai

[e]. Note that
C (θ, h, a) =
=

Z

θ(e)c(e, h, a)de.

n
ZR

Rn

αa(e) + (1 − q(a(e), h))ke − ê+ k2 dµ(e).

18

Since ai (e) + (1 − q(ai (e), hi ))kei − êi+ k2 ≥ 0, then by Theorem 2.2 in [65], one obtains that
Z
αa(e) + (1 − q(a(e), h))ke − ê+ k2 dµ(e)
n
R
Z
≤ lim inf
αai (e) + (1 − q(ai (e), h))ke − êi+ k2 dµi (e),
i→∞

Rn

which means that C (θ, h, a) is lower semicontinuous.

4.5

Relative Discounted Value Function Condition C5

Note that by Lemma 5 in [58], if
inf J (d, θ, h) < ∞,

(27)

lim sup[υβ (θ, h) − mβ ] < ∞, ∀(θ, h) ∈ S.

(28)

d,θ,h

then (19) can be equivalently written as

β↑1

To verify (27), consider a suboptimal policy, denoted by d⋄ , where at each time instant the maximal
transmission power ū is used. Given a belief θ, denote by Var(θ) the second central moment, i.e.,
Z
θ(e)(e − ê)(e − ê)⊤ de,
Var(θ) =

(29)

Rn

where ê = E[e|e ∼ θ] is the mean. Then for any initial state (θ, h) ∈ S, if the policy d⋄ is used, one can
rewrite (16) as
C (θk , hk , ak ) = αū + (1 − q(ū, hk ))Tr(Var(θk ))
and for any k ≥ 1,

 AVar(θ )A⊤ + W, if γ = 0,
k
k
Var(θk+1 ) =
 W,
otherwise,

with P(γk = 0) = 1 − q(ū, hk ) and Var(θ1 ) = AΣ0 A⊤ + W . Then for any initial state (θ, h) ∈ S, with
Assumption 4, there exists a finite upper bound κ(θ), which depends on the initial state θ, such that for
⋄

any k ≥ 1, Edθ,h [Tr(Var(θk ))] ≤ κ(θ). Then one obtains that
inf J (d, θ, h) ≤ inf J (d⋄ , θ, h)
θ,h

d,θ,h

< inf κ(θ) + αū
θ

<∞.
We now focus on the verification of (28). Define the stopping time
Tβ , inf{k ≥ 1 : υβ (θk , hk ) ≤ υ β (N0,W )},

19

where υ β (N0,W ) = minh∈h υβ (N0,W , h). Then by Lemma 4.1 in [57], one has for any β < 1 and (θ, h) ∈ S,
υβ (θ, h) − mβ ≤υ β (N0,W ) − mβ


Tβ −1
X
+ inf Edθ,h 
C (θk , hk , ak ) .
d∈D

(30)

k=1

⋄

We now prove the finiteness of Edθ,h [Tβ ] with any initial state (θ, h). To this end, let h∗ = arg minh∈h υβ (N0,W , h)
and
T∗β , inf{k ≥ 1 : (θk , hk ) = (N0,W , h∗ )}.
Note that the dependence of T∗β on β is due to h∗ . Then one can see that for any realization of {θk } and
{hk },
T∗β ≥ Tβ
always holds. Note that {hk } evolves independently. Though {θk } depends on the realization of {hk },
under the policy d⋄ , P(θk = N0,W ) ≥ q(ū, h) for all k > 1 with any initial state θ. Based on the above two
⋄

observations, we construct a uniform (for any 0 < β < 1) upper bound of Edθ,h [T∗β ] as follows. Define
K (h, h′ ) = min{k > 1 : hk = h′ , h1 = h}
as the first time hk reaches h′ when starting at h. Then given the initial state h, let {Ti }i≥1 be a sequence
of independent random variables such that E[T1 ] = E[K (h, h∗ )] and E[Ti ] = E[K (h∗ , h∗ )], i > 1. Let χ be
a geometrically distributed random variable with success probability q(ū, h). Then one obtains that
⋄

Edθ,h [T∗β ]
#
" χ
X
Ti
≤E
i=1

1
≤
max{E[K (h, h∗ )], E[K (h∗ , h∗ )]}
q(ū, h)
1
≤
max {max{E[K (h, h′ )], E[K (h′ , h′ )]}}
q(ū, h) h,h′ ∈h
< ∞,

(31)
(32)

where the second inequality follows from the Wald’s identity and the last inequality follows from the
assumption that

h is a finite set and Assumption 1-(v). Note that since (31) is independent of β, Edθ,h⋄ [T∗β ]

is uniformly bounded.
With a little abuse of notation, define
K (θ, θ ′ ) = min{k > 1 : θk = θ ′ , θ1 = θ}.

(33)

Then for any initial state (θ, h), the finiteness of J (d, θ, h) implies P (K (θ, N0,W ) < ∞) = 1. Then by the
definition of υ β (N0,W ), one obtains that
lim sup(υ β (N0,W ) − mβ ) = 0.
β↑1

20

(34)

One thus obtains that for any (θ, h) ∈ S,
lim sup[υβ (θ, h) − mβ ]
β↑1





Tβ −1

≤ lim sup inf Edθ,h 
β↑1

d∈D



X
k=1

Tβ −1

⋄

≤ lim sup Edθ,h 
β↑1

X
k=1

C (θk , hk , ak )


C (θk , hk , ak )

T∗ −1

β
X
⋄
≤ lim sup Edθ,h 
C (θk , hk , ak )
β↑1

k=1

d⋄

≤ lim sup Eθ,h [T∗β − 1](κ(θ) + αū)
β↑1

< ∞,
where the first inequality follows from (30) and (34), the last second inequality follows from the Wald’s
identity and the last inequality follows from (32). The condition (relative discounted value function) thus
is verified.
The proof of Theorem 1 now is complete.

5

Structural Description: Majorization Interpretation

In this section, we borrow the technical reasoning from [28,66] to show that the optimal transmission power
allocation strategy has a symmetric and monotonic structure and the optimal estimator has a simple form
for cases where the system is scalar.
Before presenting the main theorem, we introduce a notation as follows. For a policy d(d) ∈ Dds with
d(θ, h) = a(e), with a little abuse of notations, we write a(e) as aθ,h (e) to emphasize its dependence
on the state (θ, h). We also use aθ,h (e) to represent the deterministic and stationary policy d(d) with
d(θ, h) = a(e).
According to Theorem 1, to solve Problem 1, we can restrict the optimal policy to be deterministic and
stationary without any performance loss. The following theorem suggests that we the optimal policy can
be further restricted to be a specific class of functions.
Theorem 2. Let the system (1) be scalar. There exists an optimal deterministic and stationary policy
a∗θ,h (e) such that a∗θ,h (e) is a symmetric and monotonic function of e, i.e., for any given (θ, h) ∈ S,
(i). a∗θ,h (e) = a∗θ,h (−e) for all e ∈ R; (ii). a∗θ,h (e1 ) ≥ a∗θ,h (e2 ) when |e1 | ≥ |e2 | with equality for |e1 | = |e2 |.
The proof is given in Section 5.2. Note that Theorem 2 does not require a symmetric initial distribution
µx0 . Intuitively, this is because 1) whatever the initial distribution is, the belief state will reach the very
special state N0,W sooner or later, 2) we focus on the long term average cost and the cost incurred by
finite transient states can be omitted.
21

Remark 7. When there exists only a finite number of power levels, only the thresholds of the innovation
error used to switch the power levels are to be determined for computation of the optimal transmission
power control strategy. This significantly simplifies both the offline computation complexity and the online
implementation. The feedback of an accurate action lk in Figure 3 is possible in this scenario, since one
just need to feed back a finite number of thresholds, at which the power level switches.
In the following theorem, without a proof, we give the optimal estimator (10) when the transmission
power controller adopts the optimal policy defined in Theorem 2. The simple structure of the optimal
remote estimator gk∗ in (35) is due to the symmetric structure the function a∗θ,h (e) possesses. Recall that
τ (k) is defined in (12).
Theorem 3. Consider the optimal transmission power controller fk∗ ,
uk = fk∗ (xk , Ok− ) , a∗θk ,hk (ek )
where a∗θ,h (e) is a symmetric and monotonic function of ek . Then the optimal remote state estimator gk∗
is given by

 x ,
k
x̂k = gk∗ (Ok+ ) =
 Aτ (k) x̂

if γk = 1,
k−τ (k) ,

if γk = 0,

(35)

In the following, we focus on the proof of Theorem 2.

5.1

Technical Preliminaries

We first give some supporting definitions and lemmas as follows.
Definition 4 (Symmetry). A function f : Rn → R is said to be symmetric about a point o ∈ Rn , if, for
any two points x, y ∈ Rn , ky − ok = kx − ok implies f (x) = f (y).
Definition 5 (Unimodality). A function f : Rn → R is said to be unimodal, if there exists o ∈ Rn such
that f (o) ≥ f (o + α0 v) ≥ f (o + α1 v) holds for any v ∈ Rn and any α1 ≥ α0 ≥ 0.
Definition 6. For any given Borel measurable set B ⊂ Rn , where L(B) < ∞, we denote the symmetric
rearrangement of B by B σ , i.e., B σ is a ball centered at 0 with the Lebesgue measure L(B). For a given
integrable, nonnegative function f : Rn → R, we denote the symmetric nonincreasing rearrangement of f
by f σ , where f σ is defined as
σ

f (x) ,

Z

0

∞

1{o∈Rn :f (o)>t}σ (x)dt.

Definition 7. For any given two integrable, nonnegative functions f, g : Rn → R, we say that f majorizes
g, which is denoted as g ≺ f , if the following conditions hold:
Z
Z
σ
gσ (x)dx
f (x)dx ≥
kxk≤t

kxk≤t

22

∀t ≥ 0

(36)

and
Z

f (x)dx =

Z

g(x)dx.

Rn

Rn

Equivalently, (36) can be altered by the following condition: for any Borel set B ⊂ Rn , there always exists
R
R
another Borel set B ′ with L(B ′ ) = L(B) such that B g(x)dx ≤ B′ f (x)dx.
Recall that L, which is introduced in (14), is the saturation threshold for actions.

Definition 8 (Binary Relation R on Belief States). For any two belief states θ, θ∗ ∈ Θ, we say that θRθ∗
if the following conditions hold:
(i). there holds θ ≺ θ∗ ;
(ii). θ∗ is symmetric and unimodal about the origin point 0.
(iii). θ(e) = θ∗ (e) for any e ∈ Rn \E, where E , {e ∈ Rn : kek ≤ L} is defined below (14).
In the following, we define a symmetric increasing rearrangement of an action a ∈ A, which preserves
the average power consumption and successful transmission probability.
Definition 9. For any given Borel measurable B ⊂ Rn , where L(B) < ∞, we define
σ
Bθ,
, {e ∈ Rn : kek ≥ r},
θ̂

where θ, θ̂ ∈ Θ and r is determined such that
aσθ,θ̂ (e) ,
It can be verified that if

R

Rn \E

θ(e)de =
Z

R

R

Z

B

θ(e)de =

∞

0

Rn \E

R

Bσ

θ̂(e)de. Given an action a ∈ A, define

θ,θ̂

1{o∈Rn :a(o)>t}σ (e)dt.
θ,θ̂

θ̂(e)de, aσ (e) ∈ A. One also obtains that
θ,θ̂

a(e)θ(e)de =

Z

Rn

Rn

aσθ,θ̂ (e)θ̂(e)de

(37)

and for any h,
Z

q(a(e), h)θ(e)de =
Rn

Z

Rn



q aσθ,θ̂ (e), h θ̂(e)de.

Then the following lemma follows straightforwardly.
Lemma 2. If A is a scalar, then θRθ∗ implies φ(θ, h, a, 0)Rφ(θ∗ , h, aσθ,θ∗ , 0), where φ(·, ·, ·, ·) is the belief
update equation defined in (15).


Note that if θRθ̂, then q(a(e), h)θ(e)Rq aσθ,θ̂ (e), h θ̂(e). Then based on (37), following the same rea-

soning as in Lemma 15 in [28], one obtains the following lemma.

Lemma 3. If θRθ̂, then the following inequality about the one stage cost holds: C (θ, h, a) ≥ C (θ̂, h, aσθ,θ̂ ).

23

5.2

Proof of Theorem 2

We then proceed to prove Theorem 2 in a constructive way. To be specific, we show that for any initial state (θ, h), and any deterministic and stationary policy d(d) ∈ Dds 5 such that J (d(d), θ, h) < ∞6 ,
ˆ ∈ Dds with a symmetric and monotonic structure defined in Theothere exists another policy d̂(d)
ˆ θ, h) ≤ J (d(d), θ, h). For any initial state (θ, h) and any policy with finite cost,
rem 2 such that J (d̂(d),
P (K (θ, N0,W ) < ∞) = 1, where K (·, ·) is defined in (33). Then, without loss of generality, we can assume
that the initial state θ = N0,W . Let d(θ, h) = aθ,h (e), then under the policy d(d), the evolution of belief
states is illustrated in Figure 4. Note that since the evolution of channel gains is independent of action a,
we omit it. In Figure 4, the channel gain is assumed to be a constant h. Notice the difference between the
notations θ i and θk . The quantity θ i denotes an element in Θ, while θk is the belief state of the MDP at
time instant k.

1 − p0

1 − pi

✤✜
❘
p0 ✤✜

θ1

θ0

...

p1
❥
✣✢
✣✢
■
■
pi
■

✤✜
✤✜
❘

θi

θi+1

...

✣✢
✣✢

pi+1

Figure 4: Evolution of belief states. The special state θ 0 = N0,W , pi = ϕ(θ i , h, aθi ,h ), ∀i ≥ 0 is the successful
transmission probability, and θ i+1 = φ(θ i , h, aθi ,h , 0), ∀i ≥ 0. When the belief state is θ i , it incurs cost
C (θ i , h, aθi ,h ).
ˆ h) = âθ,h (e), and p̂i and θ̂ i be the counterparts of pi and θ i , respectively. To facilitate presentation,
Let d(θ,
let ai , aθi ,h and âi = âθ̂i ,h . Then {âi }i∈N are constructed as follows:
σ
âi = ai θi ,θ̂i .

Then by Lemmas 2 and 3, one obtains that

p̂i =pi , ∀i ≥ 0,
θ̂ 0 = θ 0 = N0,W , θ i Rθ̂ i , ∀i ≥ 1,
C (θ i , h, ai ) ≥C (θ̂ i , h, âi ), ∀i ≥ 0, h ∈ h.
ˆ θ, h) ≤ J (d(d), θ, h). Since {âi }i∈N is symmetric and increasing, and θ̂ i is
It then follows that J (d̂(d),
symmetric, one concludes the results of the theorem.

6

Conclusion and Future Work

In this paper, we studied the remote estimation problem where the sensor communicates with the remote
estimator over a fading channel. The transmission power control strategy, which affects the behavior of
5
6

By Theorem 1, without any performance loss, we can just focus on the class of deterministic and stationary policies Dds .
Note that it is impossible for a policy with infinite cost to be optimal.

24

communications, as well as the remote estimator were optimally co-designed to minimize an infinite horizon cost consisting of power consumption and estimation error. We showed that when determining the
optimal transmission power, the full information history available at the sensor is equivalent to its belief
state. Since no constraints on the information structure are imposed and the belief state is updated recursively, the results we obtained provide some insights into the qualitative characterization of the optimal
power allocation strategy and facilitate the related performance analyses. In particular, we provided some
structural results on the optimal power allocation strategy and the optimal estimator, which simplifies
the practical implementation of the algorithm significantly. One direction of future work is to explore the
structural description of the optimal remote estimator and the optimal transmission power control rule
when the system matrix is a general one.

Appendix A
We will introduce background knowledge for weak convergence of probability measures and generalized
skorohad space.

6.1

Weak Convergence of Probability Measures

Let X be a general Polish space and X be the Borel σ−field [55]. Let µ and {µi,i∈N } be probability
measures on (X , X ). By the Portmanteau Theorem [56], the following statements are equivalent:
(i). µi converges weakly to µ.
(ii). limi→∞

R

bdµi →

R

bdµ for every bounded and continuous function b(·) on X.

(iii). limi→∞ µi (B) → µ(B) for every µ−continuity set B.
w

We write as µi → µ if µi converges weakly to µ. The Prohorov metric [56] is a metrization of this weak
convergence topology. Let M be the collection of all the probability measures defined on (X , X ). If M is
endowed with the weak convergence topology, then M is a Polish space.

6.2

Generalized Skorohod Space [64].

Let (X , dX (·, ·)) be a compact metric space and Λ be a set of homeomorphisms from X onto itself. Let π
be a generic element of Λ, then on Λ, define the following three norms:
kπks = sup dX (πx, x)
x∈X

kπkt =

log

sup
x,y∈X :x6=y

kπkm =kπks + kπkt .

25

dX (πx, πy)
dX (x, y)

Note that kπkt = kπ−1 kt . Let Λt ⊆ Λ be the group of homeomorphisms with finite k · kt , i.e.,
Λt = {π ∈ Λ : kπkt < ∞}.
Note that since X is compact, each element in Λt also has finite k · km . Let Br (X ) be the set of bounded
real-valued functions defined on X , then the Skorohod distance d(·, ·) for f, g ∈ Br (X ) is defined by

d(f, g) = inf
{ǫ > 0 : ∃ π ∈ Λt such that
ǫ
kπkm < ǫ and sup |f (x) − g(πx)| < ǫ}.

(38)

x∈X

Let W be the set of all finite partitions of X that are invariant under Λ. Let I∆ be the collection of
functions that are constant on each cell of a partition ∆ ∈ W. Then the generalized Skorohod space on X
are defined by
D(X ) = {f ∈ Br (X ) : ∃ ∆ ∈ W, g ∈ I∆ such that

d(f, g) = 0}.

(39)

By convention, two functions f and g with d(a, b) = 0 are not distinguished. Then by Lemma 3.4, Theorems
3.7 and 3.8 in [64], the space D(X ) of the resulting equivalence classes with metric

d(·, ·) defined in (38)

is a complete metric space. For f ∈ Br (X) and ∆ = {δj } ∈ W, define

w(f, ∆) = max sup{|f (x) − f (y)| : x, y ∈ δj }.
δj

(40)

x,y

For f ∈ Br (X), f ∈ D(X ) if and only if lim∆ w(f, ∆) → 0, with the limits taken along the direction of
refinements.
In the end, we should remark that in our case for the action space A, the collection of homeomorphisms
Λt used to define the Skorohod distance in (38) is defined on E instead of Rn .

Appendix B
In the following lemma, we give a condition on the probability measures, under which the weak convergence
implies set-wise convergence.
Lemma 4. Let µ and {µi,i∈N } be probability measures defined on (Rn , B(Rn )), where B(Rn ) denotes the
Borel σ−algebra of Rn . Suppose they are absolutely continuous with respect to the Lebesgue measure. Then
the following holds:
w

sw

µi → µ ⇒ µi → µ,

(41)

sw

where µi → µ represents set-wise convergence, i.e., for any A ∈ B(Rn ), µi (A) → µ(A).
Proof. The Borel σ−algebra B(Rn ) can be generated by n−demensional rectangles, i.e.,
B(Rn ) = σ({(x1 , y1 ] × · · · × (xn , yn ] : xj , yj ∈ R}).
26

(42)

Since µ is absolutely continuous with respect to Lebesgue measure, all the rectangles are µ−continuity
sets. By the Portmanteau Theorem [56], for any xj , yj ∈ R,
µi ((x1 , y1 ] × · · · × (xn , yn ]) → µ((x1 , y1 ] × · · · × (xn , yn ]).
Then statement (41) follows from (42), which completes the proof.

References
[1] W. S. Wong and R. W. Brockett, “Systems with finite communication bandwidth-part II: Stabilization
with limited information feedback,” IEEE Transactions on Automatic Control, vol. 44, no. 5, pp.
1049–1053, 1999.
[2] G. N. Nair and R. J. Evans, “Stabilizability of stochastic linear systems with finite feedback data
rates,” SIAM Journal on Control and Optimization, vol. 43, no. 2, pp. 413–436, 2004.
[3] S. Tatikonda and S. Mitter, “Control under communication constraints,” IEEE Transactions on
Automatic Control, vol. 49, pp. 1056–1068, July 2004.
[4] H. Ishii and B. A. Francis, “Quadratic stabilization of sampled-data systems with quantization,”
Automatica, vol. 39, pp. 1793–1800, 2003.
[5] M. Fu and L. Xie, “The sector bound approach to quantized feedback control,” IEEE Transactions
on Automatic Control, vol. 50, no. 11, pp. 1698–1711, 2005.
[6] J. Hespanha, P. Naghshtabrizi, and Y. Xu, “A survey of recent results in networked control systems,”
Proceedings of the IEEE, vol. 95, no. 1, pp. 138–162, 2007.
[7] L. Schenato, “Optimal estimation in networked control systems subject to random delay and packet
drop,” IEEE Transactions on Automatic Control, vol. 53, no. 5, pp. 1311–1317, 2008.
[8] L. Shi, L. Xie, and R. M. Murray, “Kalman filtering over a packet-delaying network: A probabilistic
approach,” Automatica, vol. 45, no. 9, pp. 2134–2140, 2009.
[9] K. You, M. Fu, and L. Xie, “Mean square stability for kalman filtering with markovian packet losses,”
Automatica, vol. 47, no. 12, pp. 2647–2657, 2011.
[10] B. Sinopoli, L. Schenato, M. Franceschetti, K. Poolla, M. I. Jordan, and S. S. Sastry, “Kalman
filtering with intermittent observations,” IEEE Transactions on Automatic Control, vol. 49, no. 9,
pp. 1453–1464, 2004.
[11] M. Huang and S. Dey, “Stability of Kalman filtering with markovian packet losses,” Automatica,
vol. 43, pp. 598–607, 2007.

27

[12] L. Schenato, B. Sinopoli, M. Franceschetti, K. Poolla, and S. S. Sastry, “Foundations of control and
estimation over lossy networks,” Proceedings of the IEEE, vol. 95, no. 1, pp. 163–187, 2007.
[13] V. Gupta, N. C. Martins, and J. S. Baras, “Optimal output feedback control using two remote sensors
over erasure channels,” IEEE Transactions on Automatic Control, vol. 54, no. 7, pp. 1463–1476, 2009.
[14] A. Mainwaring, D. Culler, J. Polastre, R. Szewczyk, and J. Anderson, “Wireless sensor networks for
habitat monitoring,” in Proc. International Workshop on Wireless Sensor Networks and Applications.
ACM, 2002, pp. 88–97.
[15] C. Yang and L. Shi, “Deterministic sensor data scheduling under limited communication resource,”
IEEE Transactions on Signal Processing, vol. 59, no. 10, pp. 5050–5056, 2011.
[16] L. Zhao, W. Zhang, J. Hu, A. Abate, and C. J. Tomlin, “On the optimal solutions of the infinitehorizon linear sensor scheduling problem,” IEEE Transactions on Automatic Control, vol. 59, no. 10,
pp. 2825–2830, 2014.
[17] L. Shi, P. Cheng, and J. Chen, “Optimal periodic sensor scheduling with limited resources,” IEEE
Transactions on Automatic Control, vol. 56, no. 9, pp. 2190–2195, 2011.
[18] M. F. Huber, “Optimal pruning for multi-step sensor scheduling,” IEEE Transactions on Automatic
Control, vol. 57, no. 5, pp. 1338–1343, 2012.
[19] D. Shi and T. Chen, “Optimal periodic scheduling of sensor networks: A branch and bound approach,”
Systems & Control Letters, vol. 62, no. 9, pp. 732–738, 2013.
[20] S. Liu, M. Fardad, P. K. Varshney, and E. Masazade, “Optimal periodic sensor scheduling in networks
of dynamical systems,” IEEE Transactions on Signal Processing, vol. 62, no. 12, pp. 3055–3068, 2014.
[21] K. J. Åström and B. Bernhardsson, “Comparison of riemann and lebesque sampling for first order
stochastic systems,” in Proceedings of the 41st IEEE Conference on Decision and Control, vol. 2.
IEEE, 2002, pp. 2011–2016.
[22] Y. Xu and J. Hespanha, “Estimation under uncontrolled and controlled communications in networked
control systems,” in Proc. IEEE Conference on Decision and Control and European Control Conference, Dec 2005, pp. 842–847.
[23] O. Imer and T. Başar, “Optimal estimation with limited measurements,” in Proceedings of the 44th
IEEE Conference on Decision and Control and European Control Conference, December 2005, pp.
1029–1034.
[24] R. Cogill, S. Lall, and J. P. Hespanha, “A constant factor approximation algorithm for event-based
sampling,” in Proceedings of the American Control Conference, 2007, pp. 305–311.

28

[25] J. Sijs and M. Lazar, “On event based state estimation,” in Hybrid Systems: computation and control.
Springer, 2009, pp. 336–350.
[26] G. Lipsa and N. Martins, “Remote state estimation with communication costs for first-order LTI
systems,” IEEE Transactions on Automatic Control, vol. 56, no. 9, pp. 2013–2025, Sept 2011.
[27] J. Wu, Q.-S. Jia, K. H. Johansson, and L. Shi, “Event-based sensor data scheduling: Trade-off between
communication rate and estimation quality,” IEEE Transactions on Automatic Control, vol. 58, no. 4,
pp. 1041–1046, 2013.
[28] A. Nayyar, T. Başar, D. Teneketzis, and V. V. Veeravalli, “Optimal strategies for communication
and remote estimation with an energy harvesting sensor,” IEEE Transactions on Automatic Control,
vol. 58, no. 9, pp. 2246–2260, 2013.
[29] C. Ramesh, H. Sandberg, and K. H. Johansson, “Design of state-based schedulers for a network of
control loops,” IEEE Transactions on Automatic Control, vol. 58, no. 8, pp. 1962–1975, 2013.
[30] J. Wu, K. Johansson, and L. Shi, “A stochastic online sensor scheduler for remote state estimation
with time-out condition,” IEEE Transactions on Automatic Control, vol. 59, no. 11, pp. 3110–3116,
2014.
[31] A. Molin, “Optimal event-triggered control with communication constraints,” Ph.D. dissertation,
München, Technische Universität München, Diss., 2014, 2014.
[32] V. Gupta, T. Chung, B. Hassibi, and R. M. Murray, “On a stochastic sensor selection algorithm
with applications in sensor scheduling and dynamic sensor coverage,” Automatica, vol. 42, no. 2, pp.
251–260, 2006.
[33] D. Han, Y. Mo, J. Wu, S. Weerakkody, B. Sinopoli, and L. Shi, “Stochastic event-triggered sensor
schedule for remote state estimation,” IEEE Transactions on Automatic Control,to appear, 2015.
[34] S. Weerakkody, Y. Mo, B. Sinopoli, D. Han, and L. Shi, “Multi-sensor scheduling for state estimation
with event-based, stochastic triggers,” IEEE Transactions on Automatic Control, accepted.
[35] S. Yüksel and T. Başar, Stochastic networked control systems: Stabilization and optimization under
information constraints.

Springer Science & Business Media, 2013.

[36] A. Goldsmith, Wireless Communications.

Cambridge University Press, 2005.

[37] N. Elia, “Remote stabilization over fading channels,” Systems & Control Letters, vol. 54, no. 3, pp.
237–249, 2005.
[38] S. Dey, A. S. Leong, and J. S. Evans, “Kalman filtering with faded measurements,” Automatica,
vol. 45, no. 10, pp. 2223–2233, 2009.

29

[39] N. Xiao, L. Xie, and L. Qiu, “Feedback stabilization of discrete-time networked systems over fading
channels,” IEEE Transactions on Automatic Control, vol. 57, no. 9, pp. 2176–2189, 2012.
[40] D. E. Quevedo, A. Ahlén, and K. H. Johansson, “State estimation over sensor networks with correlated
wireless fading channels,” IEEE Transactions on Automatic Control, vol. 58, no. 3, pp. 581–593, 2013.
[41] C.-H. Wang and S. Dey, “Distortion outage minimization in Rayleigh fading using limited feedback,”
in IEEE Global Telecommunications Conference (GLOBECOM). IEEE, 2009, pp. 1–8.
[42] D. E. Quevedo, A. Ahlén, and J. Østergaard, “Energy efficient state estimation with wireless sensors
through the use of predictive power control and coding,” IEEE Transactions Signal Processing, vol. 58,
no. 9, pp. 4811–4823, 2010.
[43] A. S. Leong, S. Dey, G. N. Nair, and P. Sharma, “Power allocation for outage minimization in state
estimation over fading channels,” IEEE Transactions on Signal Processing, vol. 59, no. 7, pp. 3382–
3397, 2011.
[44] J. Wu, Y. Li, D. E. Quevedo, V. Lau, and L. Shi, “Data-driven power control for state estimation: a
Bayesian inference approach,” Automatica, vol. 54, pp. 332–339, 2015.
[45] K. Gatsis, A. Ribeiro, and G. J. Pappas, “Optimal power management in wireless control systems,”
IEEE Transactions on Automatic Control, vol. 59, no. 6, pp. 1495–1510, 2014.
[46] M. Nourian, A. S. Leong, and S. Dey, “Optimal energy allocation for Kalman filtering over packet
dropping links with imperfect acknowledgments and energy harvesting constraints,” IEEE Transactions on Automatic Control, vol. 59, no. 8, pp. 2128–2143, 2014.
[47] M. Nourian, S. Dey, and A. Ahlén, “Distortion minimization in multi-sensor estimation with energy
harvesting,” IEEE Journal on Selected Areas in Communications, vol. 33, no. 3, pp. 524–539, 2015.
[48] M. Fu and C. E. de Souza, “State estimation for linear discrete-time systems using quantized measurements,” Automatica, vol. 45, no. 12, pp. 2937 – 2945, 2009.
[49] A. S. Leong and S. Dey, “Power allocation for error covariance minimization in Kalman filtering over
packet dropping links,” in Proceedings of the 51st IEEE Conference on Decision and Control. IEEE,
2012, pp. 3335–3340.
[50] Y. Mostofi and R. M. Murray, “To drop or not to drop: design principles for Kalman filtering over
wireless fading channels,” IEEE Transactions on Automatic Control, vol. 54, no. 2, pp. 376–381, 2009.
[51] S. S. Haykin, Digital Communications.

New York: Wiley, 1988.

[52] B. Anderson and J. Moore, Optimal Filtering.

Prentice Hall, 1979.

30

[53] A. Nayyar, “Sequential decision making in decentralized systems,” Ph.D. dissertation, University of
California, Berkeley, 2011.
[54] A. R. Cassandra, “Exact and approximate for partially observable Markov decision processes,” Ph.D.
dissertation, Brown University, 1998.
[55] R. Durrett, Probability: Theory and Examples.

Cambridge university press, 2010.

[56] P. Billingsley, Convergence of Probability Measures.

New York: John Wiley & Sons, 1999.

[57] M. Schäl, “Average optimality in dynamic programming with general state space,” Mathematics of
Operations Research, vol. 18, no. 1, pp. 163–172, 1993.
[58] E. A. Feinberg, P. O. Kasyanov, and N. V. Zadoianchuk, “Average cost Markov decision processes
with weakly continuous transition probabilities,” Mathematics of Operations Research, vol. 37, no. 4,
pp. 591–607, 2012.
[59] O. Hernández-Lerma and J. B. Lasserre, Discrete-time Markov Control Processes: Basic Optimality
Criteria.

New York: Springer Science & Business Media, 2012, vol. 30.

[60] M. Yan, Introduction to Topology: Theory and Applications. Beijing: Higher Education Press, 2010.
[61] K. Lange, “Borel sets of probability measures,” Pacific Journal of Mathematics, vol. 48, pp. 141–161,
1973.
[62] E. A. Feinberg, P. O. Kasyanov, and N. V. Zadoianchuk, “Fatou’s lemma for weakly converging
probabilities,” Theory of Probability & Its Applications, vol. 58, no. 4, pp. 683–689, 2014.
[63] W. Rudin, Principles of Mathematical Analysis.

New York: McGraw-Hill, 1964, vol. 3.

[64] M. L. Straf, “Weak convergence of stochastic processes with several parameters,” in Proceedings of the
Sixth Berkeley Symposium on Mathematical Statistics and Probability, Volume 2: Probability Theory.
Berkeley, Calif.: University of California Press, 1972, pp. 187–221.
[65] O. Hernández-Lerma and J. B. Lasserre, “Fatou’s lemma and Lebesgue’s convergence theorem for
measures,” International Journal of Stochastic Analysis, vol. 13, no. 2, pp. 137–146, 2000.
[66] B. Hajek, K. Mitzel, and S. Yang, “Paging and registration in cellular networks: Jointly optimal
policies and an iterative algorithm,” IEEE Transactions on Information Theory, vol. 54, no. 2, pp.
608–622, 2008.

31

