Is there a name and common technique for such equations, where $A$ and $B$ are matrices and $x$ a vector?


$Ax+f(\lambda)Bx=g(\lambda)x$.



What is such an equation called?
Akin to my comment, this equation can be called a nonlinear **generalized** eigenvalue problem. Usually, $f$ and $g$ are polynomials in $\lambda$, but more general nonlinearities might be allowed. In general, I doubt there will be robust, globally convergent method for this equation that gets all the solutions. The [talk](http://www.maths.dur.ac.uk/events/Meetings/LMS/2008/CLAPDE/Talks/mehrmann1.pdf) or [this paper](http://issnla2010.ba.cnr.it/MehV04.pdf) might be good starting points (see especially the paper).


