During QE what prevents the banks from selling government debt to invest in the stock market instead of lending out the money? This is how QE is supposed to work:

The fed buys government debt from banks.
This lowers the long term interest rates and businesses and consumers borrow more, which is good for the economy.
But what if the banks instead of lending out money put most of their money in the stock market?
They know that somewhat later the money will get to the general population.
And some of the general population make a good living. So good in fact that they save a lot of money in index funds, and in particular when the interes rates are low. The banks already know in advance that this will drive up the stock market.
Since they can buy the stocks before the money has reached the general population they can buy the stocks at a lower price.
As a more detailed example assume the investment bank G. Gecko own 1e6 10 years treasury bonds at USD 1000 per bond.
The bonds have a coupon rate of 10 USD, resulting in a yearly return of 20 USD = 2%. Now the Fed starts buying up 10 year treasury bonds and the price of these rises to 2000 USD. Now the return rate of these bonds are only 1%. This is not enough so G. Gecko sells it bonds. Now it has doubled its money but it need to invest these somewhere. So it buys Apple stocks.
Another bank also sell treasury bonds but use the new money to lend out.
This improves the economy and some of the general population gets more money.
Some of this they use to buy Apple stocks. However this drives up the price of the Apple stock. Let us say it doubles. Now G. Gecko have turned their initial 1e9 USD into 4e9 USD. Apple does not benefit directly from the rise in its stock price? The general population seem to benefit from the money trickling down from the other bank but the big winner here seems to be G. Gecko.
During QE what prevents the banks from selling government debt to invest in the stock market instead of lending out the money?

Nothing. Depending on which country we are talking about there might be a regulation preventing/limiting some banks investing in a stock market, in order to make banking sector less prone to collapse. For example, Basel rules mandate that banks have to keep certain level of capital buffer for their investments in stock market in order to make them resilient (see discussion about that here). This is very context dependent, there is actually no such thing as just a bank, there are commercial banks, investment banks, retail banks etc - all are regulated slightly differently and it is beyond scope of an SE answer to provide detailed overview rules for all individual institutions.
Besides that the only thing that would restrict them from doing so is that at some point it would not be profitable, especially if there is sufficient demand for loans.  Both equity and credit prices are subject to supply and demand forces.

But what if the banks instead of lending out money put most of their money in the stock market?

Ceteris paribus, this will stimulate economic activity the same way as issuing a loan would. From a macro perspective there is a little difference between bank buying from a firm A a bond with face value 1000USD with 10% coupon rate, versus bank buying 1000USD worth of stock expected to pay 100$ yearly dividend. What stimulates economic activity is firms getting the money (no matter whether as a loan or in exchange for equity) which allows them to increase investment.
If anything buying equity could potentially stimulate economy more, since there is some evidence that debt overhangs make firms invest less (e.g. see this Fed St. Louis explainer). Consequently, in situation where firms are already overleveraged such action would likely stimulate economic activity even more than just increase in borrowing.