If $X,Y$ are vector fields and $\def\Fl{\operatorname{Fl}}\Fl^X\_t$ and $\Fl^Y\_t$ their local flows, let $[\Fl^X\_t,\Fl^Y\_t]:= \Fl^Y\_{-t}\Fl^X\_{-t}\Fl^Y\_t\Fl^X\_t$ denote the group commutator of the flows. Then  

$$
\partial\_t|\_0 [\Fl^X\_{t},\Fl^Y\_{t}] =0, \qquad
\frac12 \partial\_t^2|\_0 [\Fl^X\_{t},\Fl^Y\_{t}] = [X,Y].
$$
See


* Markus Mauhart, Peter W. Michor: Commutators of flows and fields. Archivum Mathematicum (Brno) 28,3-4 (1992), 228–236, [(pdf)](http://www.mat.univie.ac.at/~michor/liebrack.pdf)


for an extension of this to formal bracket expressions of arbitrary length.


Where did Sophus Lie write this? And where did he compute something like
$$
\lim\_n \frac1n [\Fl^X\_{1/n},\Fl^Y\_{1/n}]^n\quad ?
$$



Where did Sophus Lie write the group commutator for two one parameter groups
Looking around on the internet, I found an English translation of Lie's 1891 paper *Die Grundlagen für die Theorie der unendlichen kontinuierlichen Transformationsgruppen. I.* (I.e., The foundations of the theory of infinite continuous transformation groups - I) at the location 


<http://neo-classical-physics.info/uploads/3/0/6/5/3065888/lie-_infinite_continuous_groups_-_i.pdf>


In this English translation (I haven't gone to look for the original German), Lie gives the formula for the commutator of the flows of two vector fields in the form that you require. Look at Paragraphs 35-37, noting especially the displayed equation (45) and the first displayed equation in Paragraph 36, which is exactly the first formula you asked about.


It's quite possible that these formulae appeared even earlier in Lie's papers. He mentions, in the introduction, a paper of 1883 that might well also have these formulae. I'll check when I get the chance.


Concerning the Trotter formula (your second formula), I have no idea.


