For finite groups $G$ of odd order, as $x \mapsto x^2$ is bijection (but no automorphism in general) then, we can define for each $g \in G$ the element $x^{1/2}$ by requiring $(x^{1/2})^2 = x$. Then with the new operation
$$
 x\circ y := x^{1/2} y x^{1/2}
$$
we get a power-associative [loop](https://en.wikipedia.org/wiki/Quasigroup) satisfiying the relations
$$
 x\circ (y \circ (x \circ z)) = (x\circ (y\circ x))\circ z
$$
and
$$
 (x\circ y)^{-1} = x^{-1} \circ y^{-1}.
$$
If the first is satisfied, such loops are also called *(left) Bol loops*, and if both additional equations hold they are called *(left) Bruck loops*. These definitions go back to ideas of [G. Glauberman: On loops of odd order](http://www.sciencedirect.com/science/article/pii/0021869364900171). Now I want to know, under what conditions for $G$ to we indeed have associativity? As $1 \in G$ is also neutral for the new operation, and also every inverse is also an inverse in the new operation, the question is under what conditions is $(G, \circ)$ also a group?



For a new operation on a finite group of odd order giving a loop structure, when does this also gives a group
As I mentioned in comments, I believe that H. Bender has used this operation in special situations(in particular for certain groups of odd order of nilpotence class 2) , and that his use of it appears in the book "Topics in Finite Groups" by T. Gagen ( published by LMS as I recall). I don't remember all details but here is a proof that when $G$ is a nilpotent group of (finite) odd order and nilpotence class 2, then we always have $x \circ y = y \circ x$.


Recall that if $G$ has nilpotence class 2, then we have $[xy,z] = [x,z][y,z]$ for all $x,y, z \in G$ as all commutators in $G$ are central. In particular, $[x,y]^{2} = [x^{2},y] = [x,y^{2}]$ for all $x,y \in G$. Also, we have always have $[x,y]^{-1} = [y,x]$.


Since when $G$ has odd order, we know that $x^{\frac{1}{2}}$ is an (integer) power of $x$, and likewise for $y$, we have both 
$[x^{\frac{1}{2}},y]^{2} = [x,y]$ and $[x,y^{\frac{1}{2}}]^{2} = [x,y]$, so we conclude that we always have $[x^{\frac{1}{2}},y] = [x,y^{\frac{1}{2}}] = [x,y]^{\frac{1}{2}}.$ 


Now from the definition of $x \circ y$, we may write $x \circ y = xy[y,x^{\frac{1}{2}}]$, and likewise, we may write $y \circ x = yx[x,y^{\frac{1}{2}}]$.


Now all commutators are central in $G$, so we now have 
$$(x \circ y)^{-1}(y \circ x) = [x^{\frac{1}{2}},y]y^{-1}x^{-1}(yx)[x,y^{\frac{1}{2}}] = [x^{\frac{1}{2}},y]^{2}[y,x] = [x,y][y,x] = 1,$$
so that $x \circ y = y \circ x$, as claimed.


Later edit: Actually, I think I can reconstruct the proof of associativity in this case:
We have $x \circ (y \circ z) = x(y\circ z)[y \circ z,x]^{\frac{1}{2}}$ from above.


Again recalling that commutators are central in $G$, we may continue to obtain
$$ x \circ (y \circ z) = x(yz)[z,y]^{\frac{1}{2}}[yz,x]^{\frac{1}{2}},$$ where we make use of the fact that $[[z,y^{\frac{1}{2}}],x] = 1$.


Hence $$x \circ (y \circ z) = x(yz)[z,y]^{\frac{1}{2}}[y,x]^{\frac{1}{2}}[z,x]^{\frac{1}{2}}$$.


Similarly (and making use of commutativity), we obtain $$(x \circ y) \circ z = z \circ (x \circ y) = z(xy)[y,x]^{\frac{1}{2}}[x,z]^{\frac{1}{2}}[y,z]^{\frac{1}{2}}$$.


Taking the product of the inverse of the first expression with the second expression ( and recalling that commutators are central), we obtain
$(yz)^{-1}x^{-1}zxy [x,z][y,z] = [z,xy][xy,z] = 1$. Hence $\circ$ is associative.


Later edit: In general, given that $(G,\circ)$ is a group, it is in fact necessary that $G$ (with its original operation) should be nilpotent, though I make no attempt to bound the nilpotency class. We have seen that if $G$ has nilpotence class $2$ (or less) then $(G,\circ)$ is a group.


Suppose then that our $G$ of odd order is such that $(G,\circ)$ is a group ( but we make no further assumptions on $G$ with respect to its original operation). Then $(G,\circ)$ is in fact an Abelian group, as noted in comments for we are told that $(x \circ y)^{-1} = x^{-1}\circ y^{-1}$ (which can also be verified by direct computation), while we have $(x \circ y)^{-1} = y^{-1} \circ x^{-1}$ if $\circ$ is associative.


Thus we have $x^{\frac{1}{2}}yx^{\frac{1}{2}} = y^{\frac{1}{2}}xy^{\frac{1}{2}}$ for all $x,y \in G$. As M. Farrokhi D.G. notes in comments below, this is equivalent to requiring that $x^{\frac{1}{2}}y^{\frac{1}{2}}$ and $y^{\frac{1}{2}}x^{\frac{1}{2}}$ commute for all $x,y \in G$. Since squaring is a bijection on $G$ this is equivalent to $[xy,yx] = 1$ for all $x,y \in G$. Thus y and $x^{-1}yx$ commute for all $x,y \in G$. Thus $[[y,x],y] = 1= [[x,y],y] = 1$ for all $x,y \in G$. Thus $G$ is an Engel group. Engel groups are well-studied and finite Engel groups are known to be nilpotent (apparently credited to Zorn), though there are examples of infinite Engel groups which are not nilpotent.
Hence $G$ is nilpotent with respect to its original group operation.


Even later edit: I will now address the question of nilpotence class, making use of a survey article by Gunnar Traustason, which is available online. The group $G$ is a $2$-Engel group, as seen above. Since we now know that $G$ is nilpotent, $G$ decomposes as a direct product of its Sylow $p$-subgroups, and clearly $(G,\circ)$ does too, in a way which respects the decomposition for $G$.


Hence it suffices to understand the situation when $G$ is a $p$-group for some odd prime $p$. It was in fact (at least implicitly) proved by W. Burnside that a $2$-Engel group of order prime to $3$ has nilpotency class $2$ ( and we have already seen that for a group $G$ of nilpotency class $2$, $(G,\circ)$ is indeed a group). It was proved somewhat later that a $3$-group which is a $2$-Engel group has nilpotency class at most $3$. Hence the only outstanding question at this point is whether there is a $3$-group $G$ of nilpotency class $3$ such that $(G,\circ)$ is a group. For we know that a group $G$ of odd order such that $(G,\circ)$ is a group is a direct product of a $3^{\prime}$-group which is nilpotent of class at most $2$ with a $3$-group which is nilpotent of class at most $3$.


Final edit: In fact, I think it already follows from Lemma 7 of the paper of Glauberman mentioned in the statement of the question that if $(G,\circ)$ is a (necessarily Abelian as remarked before) group, then $G$ must be nilpotent of class $2$.


