Most (stochastic) "gradient descent" type algorithms (such as Nesterov-accelerated gradient-descent or ADAM) seem to be well-defined only for functions which are either:


* lower semi-continuous + convex (these sub-gradient methods),
* non-smooth but locally-lipschitz (Clarke's generalized methods),


The first of these scenarios may admit discontinuities but not the later. So I'm left wondering:


*Are there "generaled (stochastic?) gradient-descent type algorithms which can optimize non-smooth and non-convex functions?*



Gradient-descent "type" Methods for non-convex and non-smooth functions
Several splitting methods fit the bill: Often the non-convexity and the non-smoothness come from different parts of the objective and one can split the objective like $ f(x)=g(x) +h(x)$ with a convex but non-smooth $g$ and a non-convex but smooth $h$. In this case one can, for example, try a proximal gradient method which iterates
$$
x^{k+1 } = prox\_{tg}(x^k - t h'(x^k)) 
$$
and $prox\_{tg}(x) = argmin\_y tg(y) + \|y-x\|^2/2$ is the proximal mapping and $t$ is the stepsize. Under some assumptions this can be a descent method and can be shown to converge to critical points. Actually, the proximal mapping is sometimes even defined (and easily computed) for non-convex functions and the method may still be used. There is a lot more to try (going stochastic, split into more terms, try momentum or Nesterov acceleration...). Also, one there are methods that use proximal mappings of all splitted terms (if they are easy to compute) and one could also try to use subgradients instead of gradients of $h$ in case it is not smooth but the proximal mapping is not an option.


