My question is as follows:


It is known that a closed smooth curve in $\mathbb{R}^2$ is convex iff its (signed) curvature has a constant sign. I wonder if one can characterize smooth convex cones in $\mathbb{R}^3$ in a similar way.


Here is the precise statement. I say that an open set $V$ is a cone if $tV \subset V$ for any $t>0$. I say that a cone is smooth if its boundary $\partial V$ is non-empty and, say, $C^\infty$ (except at the origin). So I can define the *mean curvature* $H$ on the smooth part of the boundary of $V$. 



> 
> Do we have the equivalence between:
>  (1) $V$ is convex,
>  and
>  (2) $H$ has a constant sign on $\partial V$?
> 
> 
> 



The sign of the mean curvature on convex cones in three dimensions
The second fundamental form $II$ of $\partial V$ vanishes in the radial direction (along any line that goes through the origin), so one eigenvalue of $II$ is zero.
There are only two eigenvalues since $\dim(\partial V)=2$ and the mean curvature is (up to a factor of 2) the trace of $II$.
Therefore the nonzero eigenvalue of $II$ has the same sign as the mean curvature.


We have thus that $II$ is positive semidefinite iff the mean curvature is positive (and similarly for negative).
Positive semidefiniteness of $II$ is equivalent with convexity, so the answer to your question is yes.


Be careful with the last assertion, though.
For a submanifold on a general Riemannian manifold positive semidefiniteness of $II$ is not enough to guarantee convexity in the geodesic sense.
Convexity does always imply $II\geq0$.


Let's see why things work in our case.
Let $\gamma:[a,b]\to\mathbb R^3\setminus\{0\}$ be any line segment, and suppose $\gamma(a),\gamma(b)\in V$.
Let $\phi:\mathbb R^3\setminus\{0\}\to S^2$ be the projection to the sphere, $\phi(x)=x/|x|$.
Now $\phi\circ\gamma:[a,b]\to S^2$ is an arc of a great circle on the sphere.
The second fundamental form on $\partial V$ is positive semidefinite and vanishes in the radial direction, so $II$ on $\partial\phi(V)$ is also positive semidefinite.
It is an easier exercise to see that this implies that $\phi(V)$ is convex (in the sense that a minimal geodesic joining any two points in the set stays in the set).
Thus $\phi\circ\gamma$ only takes values in $\phi(V)$.
Since $V$ is conical, this means that $\gamma$ only takes values in $V$.


