How do economists model VNM-rationality violation? This question concerns the need to generalise utility maximisation, the fact that it's a special case of a general problem familiar to physicists, and the question of whether economists have affected a similar generalisation or instead taken a different approach. A precise statement of my question is bold below, after some theoretical exposition.

A common model of preferences in economics is that agents choose to maximise the mean of some utility function. If the utility $U\left(x,\,\phi\right)$ depends on a random variable $x$ (possibly containing multiple parameters $x^\mu$) of pdf $f$ and a user-chosen $\phi$ (on which $f$ may or may not depend), we seek to minimise $S:=\int dx\mathcal{L},\,\mathcal{L}:=-Uf$. (I'm using a physicist's notation, but this sort of mathematical problem is common to both fields.) If only $x$ were known (viz. $f=\delta\left( x-x_0\right)$), $\phi$ could be chosen to maximise $U$; the $\phi$ which does this is in general a function of $x$. Generalising to random $x$, the solution is given by $0=\frac{\partial\mathcal{L}}{\partial\phi}$, an algebraic equation that can in principle be rearranged to express $\phi$ in terms of $x$.

Maximising mean utility is equivalent to having preferences satisfying the von Neumann-Morgenstern axioms, but some empirical results such as the Ellsberg urn violate these. One possible explanation is that Knightian uncertainty bothers people, i.e. people prefer investment opportunities with known $f$. (In particular, each of Ellsberg's observed preferences was for a known probability distribution over an unknown one, even though whatever the unknown distribution was at least one of the resulting preferences would produce lower mean utility than the alternative chosen over it.) Whatever the explanation, we apparently cannot assume agents minimise $S$ as defined above.

However, since new models should acknowledge the successes of old ones as much as their failures, a model in the above vein is warranted. When physicists work with such problems, typically $L$ also depends on $\partial_\mu\phi:=\frac{\partial\phi}{\partial x^\mu}$. The $S$-minimising result is then $0=\frac{\partial\mathcal{L}}{\partial\phi}-\partial_\mu\frac{\partial\mathcal{L}}{\partial_\mu\partial\phi}$ (the last term sums over $\mu$), which unlike the previous result is a differential equation. So my question is: Have economists taken this approach? If not, what have they done instead? In particular, I'm looking for a generalisation of $\int dx Uf$ maximisation.

Let me try to make this more concrete. Suppose I'm investing in crops. Then $\phi$ could denote my portfolio, $x$ could denote future weather, $f$ would characterise the long-term climate, and ongoing climate change may present such uncertainty as to result in something analogous to an Ellsberg paradox. In particular my $\mathcal{L}$ might also depend on $\partial_\mu\phi$, my chosen portfolio's sensitivity to a guess about future weather. For example, I might have $\mathcal{L}=-Uf+a\partial_\mu\phi\frac{\partial\left(Uf\right)}{\partial_\mu\phi}$ for a constant $a$. This is just a hypothetical example, since I've not found any economic literature on this.
It appears you are looking for literature on Ambiguity Aversion and/or "Uncertainty Aversion".
You can start by looking up the work of L.G Epstein, I. Gilboa, and  D. Schmeidler.

It is an attempt to formalize the behavior of agents exhibiting behavior consistent with the Ellsberg paradox.