Let $A$ and $B$ be two self-adjoint, positive definite Compact operators on a Hilbert space $\mathcal{H}$. Further, let $A$ be trace class. Define $C\_n \equiv AB(\frac{I}{n} + BAB)^{-1}$. Does $\frac{tr(C\_nC\_n^{T})}{n}$ converge to $0$ as $n \to \infty$. The answer is affirmative when $A$ and $B$ commute as can be found in section 7.1 of the following paper-
<https://arxiv.org/pdf/1103.2692.pdf>



Convergence in trace
Yes, this is true. Denote $Q\_n=\sqrt{n}\sqrt{A}B$. We have to prove that $c\_n:=tr(\sqrt{A}Q\_n(I+Q\_n^\*Q\_n)^{-2}Q\_n^\*\sqrt{A})\to 0$. We have $c\_n=tr(AQ\_n(I+Q\_n^\*Q\_n)^{-2}Q\_n^\*)$. We need a


**Lemma.** For any compact operator $Q$ we have $Q(I+Q^\*Q)^{-2}Q^\*\leqslant \frac14I$. Moreover, for any vector $x$ we have $\|Q\_n(I+Q\_n^\*Q\_n)^{-2}Q\_n^\* x\|\to 0$.


**Proof.** We may choose orthonormal bases $(v\_i)$ and $(u\_i)$ and positive numbers $\lambda\_i$ so that $Qx=\sum \lambda\_i (x,u\_i)v\_i$, $Q^\*x=\sum \lambda\_i (x,v\_i)u\_i$, then $(v\_i)$ is eigenbasis for $Q(I+Q^\*Q)^{-2}Q^\*$ with eigenvalues $\lambda\_i^2/(1+\lambda\_i^2)^2\leqslant 1/4$. For $Q=Q\_n$ bases $v\_i,u\_i$ are fixed, $\lambda\_i=\sqrt{n}\mu\_i$ for some fixed positive numbers $(\mu\_i)$, thus each specific eigenvalue $n\mu\_i^2/(1+n\mu\_i^2)^2$ tends to 0. Representing $x$ as a sum of small vector and a finite linear combination of $v\_i$ we get the second claim. 


Now let $e\_i$ be an eigenbasis for $A$, $Ae\_i=t\_ie\_i$, $\sum t\_i<\infty$. We have $$c\_n=\sum t\_i (Q\_n(I+Q\_n^\*Q\_n)^{-2}Q\_n^\*e\_i,e\_i),$$
where $i$-th term always does not exceed $t\_i$ and each specific term tends to 0. So we may simply for given $\varepsilon>0$ choose $N$ so that $\sum\_{i>N} t\_i<\varepsilon$, estimate the tail of (1) for $i>N$ as $\varepsilon/4$ and say that first $N$ summands are less than $\varepsilon/4$ provided that $n$ is large enough. Totally $|c\_n|<\varepsilon/2$ for large $n$, as desired. 


