Consider the following controllable and observable linear system
 $$\dot x=Ax+Bu, y=Cx,$$
where $x\in \mathbb{R}^n, u\in \mathbb{R}^m, y\in \mathbb{R}^p$.
The observability of $(A,C)$ tells us that Cx(t)=0 implies x(t)=0 for any trajectory of the system. Considering this, I have the following questions:


1. Why not design a controller to steer the state to $\{x|Cx=0\}$ instead of $\{0\}$?
2. If this idea is ok, how to design such kind of controllers?


I posted it here and hope to get some answers or references.



Control a linear system to the kernal space of the output matrix.
In linear geometric control theory, the problem you posed is called the *output stabilization problem (OSP)*. See Wonham's book on linear geometric control.


The approach to solving OSP, conceptually, is as follows.


Consider the LTI system: $\dot x = A x + B u, y=Cx$. We wish to design a state feedback such that for all initial conditions, the solution $x(t)$ has the property that $y(t) = C x(t) \to 0$ as $t \to \infty$.


**Intuition:** The idea is to try to asymptotically stabilize the subspace $\ker C$, rather than the equilibrium $x=0$. However, one can show that a necessary condition for a subspace to be asymptotically stable for the closed-loop system is that the subspace be an invariant set for the closed-loop system. So we can only hope to stabilize a subspace of $\ker C$ which can be made invariant via feedback. In fact, the best thing we can do is to stabilize the *largest* subspace of $\ker C$ which can be made invariant via feedback. In geometric control theory, this is called the maximal $(A,B)$-invariant subspace contained in $\ker C$. In nonlinear circles, this is called the *zero dynamics manifold*.


**Solution**:


1) Find the maximal $(A,B)$-invariant subspace contained in $\ker C$. Let's call it $\cal V$.


2) By definition of $(A,B)$-invariant subspace, there exists a gain $K$ such that $\cal V$ is $A+BK$-invariant, i.e., $(A+BK)\cal V \subset \cal V$.


3) Set $u= Kx+v$. We obtain a new control system $\dot x = \hat A x + Bv, y=Cx$, where $\hat A :=A+BK$ and, by construction, $\ker C$ is $\hat A$-invariant.


4) Decompose the system with respect to the invariant subspace: find a coordinate transformation $(z\_1,z\_2) = T x$ such that 


(a) In $(z\_1,z\_2)$-coordinates the subspace $\cal V$ is the $z\_1$ plane, $\{(z\_1,z\_2):z\_2=0\}$.


(b) The matrix $T \hat A T^{-1}$ has the upper triangular structure
$$
T \hat A T^{-1} = \left[\matrix{A\_{11} & A\_{12} \\ 0 & A\_{22}}\right].
$$


5) Then, letting $\operatorname{col}(B\_1,B\_2) = TB$, we have the control system
$$
\matrix{\dot z\_1 = A\_{11} z\_1 + A\_{12} z\_2+ B\_1 v \\ \dot z\_2 = A\_{22} z\_2 + B\_2 v}$$


and the control objective is to stabilize the subspace $\{z2=0\}$, i.e., the equilibrium of the $z\_2$-subsystem. This is a familiar equilibrium stabilization problem. If the pair $(A\_{22},B\_2)$ is stabilisable, then there exists a feedback $v = K\_2 z\_2$ that stabilizes the subspace $\{z\_2=0\}$, which in original coordinates corresponds to stabilizing $\cal V$.


6) The final feedback solving OSP is 
$$
u=Kx + [\matrix{0 & K\_2}] T^{-1}x.$$


**Comments:**


The steps provided above are conceptual. The actual computations you have to conduct to perform each step are standard and you can find them in Wonham' book. A simpler presentation is given in "Control Theory for linear systems" by Trentelman and others.


To check whether the pair $(A\_{22},B\_2)$ is, say, controllable, you don't need to perform the steps above. You check it directly using the following 


**Theorem:** The pair $(A\_{22},B\_2)$ in step 5 is controllable if and only if


$$
{\cal V} + \operatorname{image}([\matrix{B & AB & \cdots & A^{n-1} B}]) = \Re^n. 
$$


An analogous result for stabilizability is given in Wonham.


