1. If
$$\min x'Qx + Rx$$
$$Ax\leq b$$
$$x\in\mathbb Z^n$$
is a quadratic program with $x'Qx$ is convex is there a polynomial time algorithm for this if $A$ is totally unimodular?
2. In particular if we ask that the smallest $x'Qx$ that touches on a vertex of $AX\leq b$ is that in $\mathcal P$?


We know that it is in $\mathcal P$ if $Q=0$. We also know for trivial reasons it will always first touch a vertex point.



> 
> Is it the same for convex case?
> 
> 
> If not in what cases can be find minimum $x'Qx$ that first touch a vertex point in $\mathcal P$?
> 
> 
> 



Convex integer programming on totally unimodular polytope?
This sort of problems are NP-hard even if $A=0$. For instance integer least squares <https://web.stanford.edu/~boyd/papers/pdf/int_least_squares.pdf> 


Your other question (the optimal vertex) is just as hard - consider binary least squares (ie. the restrict to the unit cube).


