Let $G$ be a locally compact Hausdorff topological group whose underlying abstract group is residually finite. Let $H\subset G$ denote the intersection of all finite-index, **closed** subgroups. Is there an example of such a $G$ where $H$ is not trivial? Is there an example where $H=G$ (i.e., every finite-index subgroup is dense)? 


My motivation for asking this question comes from the study of automorphism groups of connected, locally finite graphs. Therefore, the closer the example is to being of this type the better. E.g., an example which is an automorphism group of some structure on $\mathbb{N}$, say a hypergraph structure, would be extremely interesting; an example where $G$ is separable would be more useful than an example with cardinality larger than the continuum, and so on.



Residually finite group with dense finite index subgroups
Yes it exists (at least the weaker version). Namely, here is a way to produce second-countable abelian, totally disconnected locally compact groups whose underlying abstract group is residually finite, but in which the intersection of finite index open subgroups is not trivial.


Fix $p$ prime and an infinite countable set $I$. Let $\mathbf{Z}\_p$ be the $p$-adic group. Let $A$ be the divisible closure of $K=\mathbf{Z}\_p^I$ in $\mathbf{Q}\_p^I$ (i.e., those sequence with bounded denominator) and endow $A$ with the group topology making $K$ a compact open subgroup (with its usual product topology). The example will be a suitable subgroup of $A$ containing $K$.


Lemma: *let $F$ be a finite subset of $A$ and $\bar{F}$ its image in $A/K$. Let $U$ be a nonempty open subset of $K$, disjoint from $\langle F\rangle+pK$, and $n\ge 1$. Then there exists $x\in A$ whose image $\bar{x}$ in $A/K$ has order $p^n$, such that $\langle \bar{x}\rangle\cap \bar{F}=\{0\}$, and such that $p^nx\in U$.*


Proof: choose $z\in U-pK$ (this is possible: as $I$ is infinite $pK$ has empty interior in $K$) and set $x=p^{-n}z$. Then $\bar{x}$ has order $p^n$.
If $\langle \bar{x}\rangle\cap \bar{F}=\{0\}$ fails, then $p^\ell\bar{x}\in\bar{F}$ for some $\ell$ with $1\le\ell<n$. Hence we can write $p^\ell x=s+k$ with $s\in F$ and $K\in K$. Thus $z=p^{n-\ell}s+pk'$, with $k'=p^{n-\ell-1}k\in K$, contradiction.


Now let $(K\_n)\_{n\ge 0}$ be an enumeration of clopen subsets of $K$. Define by induction a sequence $(x\_n)$: define $F\_n=\{x\_i:i<n\}$. If $K\_n\cap (\langle F\_n\rangle+K)$ is nonempty set $x\_n=0$. Otherwise, choose $x=x\_n$ as in the lemma (for $U=K\_n$). 


Let $B$ be the (open) subgroup of $A$ generated by $K$ and $\{x\_n:n\ge 0\}$. I claim it satisfies the required properties.


(a) $B$ is residually finite as abstract group. Indeed, since it is a torsion-free abelian group, residually finite means it does not contain any isomorphic copy of $\mathbf{Q}$. If we had such a copy, it could not be contained in $K$ (which is residually finite), hence would have nontrivial projection on $B/K$. 
By construction, the cyclic subgroups $\bar{x\_n}$ generate their direct sum. So $B/K$ is also residually finite. We get a contradiction.


(b) $B$ is not residually (discrete finite) as topological group. Namely, let us check that ($\sharp$) $K=\bigcap\_n \overline{n!B}$. This implies that in any discrete quotient of $B$, the image of $K$ belongs to every finite index open subgroup, and in particular, $K$ is contained in every finite index open subgroup of $B$.


Let us prove the assertion ($\sharp$). Since $B$ is a $\mathbf{Z}\_p$-submodule of $A$, $B$ is $p$-divisible for every prime $\neq p$. So we only have to prove that $K=\bigcap\_n \overline{p^nB}$. Indeed, let $U$ be some nonempty open subset of $K$, and let us check that $p^nB\cap U$ is not empty. 
Suppose by contradiction the contrary: $p^n B\cap U$ is empty. So for every $m\ge n$, we have $p^n(K+\langle F\_m\rangle)\cap U$ empty, and hence $p^m(K+\langle F\_m\rangle)\cap U$ is empty as well. For some $m\ge n$, we have $K\_m\subset U$. Since $p^m(K+\langle F\_m\rangle)\cap U$ is empty, by construction $x\_m$ is nonzero and $p^mx\_m\in K\_m$, so $p^mB\cap U$ is non-empty, and in turn $p^nB\cap U$ is non-empty, contradiction.




---


Notes:


a) if $G$ is a locally compact group and not totally disconnected, then it has a 1-parameter subgroup, hence its underlying abstract group is not residually finite. So examples will be totally disconnected.


b) I'm not sure whether the above example can be modified to satisfy the stronger conclusion ($\neq \{0\}$ and no proper open subgroup of finite index)


c) I don't know how to find a compactly generated example. It certainly can't be abelian: indeed a compactly generated LCA-group either maps onto $\mathbf{Z}$ with open kernel, or is profinite. In both case, unless $\{0\}$, it has a proper open subgroup of finite index.


